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Abstract 

The  Department  of  Defense  (DoD)  requires  the  ability  to  quantiliably  measure 
progress  in  arenas  that  are  complex  and  difficult  to  measure,  such  as  the  stability  of  a 
region.  Therefore,  the  DoD  works  diligently  to  predict  the  effect  of  operations  and 
sponsors  research  to  improve  prediction  and  analysis.  They  desire  a  repeatable, 
systematic  methodology  to  aid  in  the  selection  of  courses  of  action  (COA)  that  efficiently 
meet  stated  objectives  and  quantitatively  measure  the  degree  of  accomplishment  of  these 
objectives.  The  author  proposes  a  value-focused  thinking  (VFT)  decision  analysis  (DA) 
approach  to  this  problem.  This  methodology  not  only  aids  in  selection  of  possible  COAs, 
but  provides  a  framework  to  compare  the  effectiveness  of  implemented  actions  via  key 
indicators.  Due  to  the  complex  nature  of  COA  selection  and  assessment,  weights  within 
the  DA  model  are  often  fluid.  Sensitivity  analysis  provides  the  justification  of  COA 
selection  in  such  an  environment.  This  thesis  focuses  on  conducting  further  analysis  of 
the  ranked  alternatives  through  a  robust  sensitivity  analysis  technique. 

Sensitivity  analysis  begins  with  the  examination  of  the  top  ranked  alternative  by 
varying  one  weight  at  a  time,  one-way  sensitivity.  The  author  then  proposes  a  more 
robust  examination  of  multiple  weight  sensitivity  using  five  unique  measures  and 
optimization  via  linear  and  non-linear  programming.  The  measures  reveal  the 
alternatives  sensitive  to  small  simultaneous  variations  of  multiple  weights  within  the 
model,  n-way  sensitivity.  Small  measure  values  indicate  sensitive  alternatives,  and 
indicate  to  a  field  commander  where  to  more  closely  examine  the  consequences  of  a 
selected  COA. 
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ROBUST  SENSITIVITY  ANALYSIS  OF  COURSES  OF  ACTION  USING  AN 


ADDITIVE  VALUE  MODEL 

I.  Introduction 

Increasingly,  the  military  is  using  decision  analysis  techniques  to  support 
commanders  in  selecting  Courses  of  Action  (COAs).  Many  of  these  approaches  assume 
additive  linear  weights.  This  thesis  examines  the  decision  sensitivity  to  variations  of 
those  weights.  In  particular,  it  examines  the  robustness  of  a  selected  solution  with  respect 
to  multiple  weight  changes  in  an  additive  value  model.  This  section  examines  the  nature 
of  military  COA  selection,  current  doctrine  and  planning  processes,  and  planned 
enhancements.  The  extensiveness,  timeliness,  and  fluidity  of  COA  selection  continue  to 
challenge  the  military  commander’s  ability  to  make  informed  decisions. 

I.A.  Background 

A  commander’s  desire  to  make  decisions  with  conflicting  objectives  dates  back  at 
least  as  far  as  the  sixth  century  B.C.  to  the  days  of  Sun  Tzu  (Wu,  2004:397).  According 
to  Griffith’s  translation  of  Sun  Tzu’s  The  Art  of  War,  a  commander  should,  “...determine 
the  enemy’s  plans  and. .  .know  which  strategy  will  be  successful  and  which  will  not” 

(Tzu,  6th  cent.  B.C. :  152).  In  2002,  the  United  States  Air  Force  Scientific  Advisory  Board 
defined  predictive  battlespace  awareness  as  the  desire  to  thoroughly  know  and  understand 
an  adversary  (SAB-TR-02-01,  Vol.  1,  2002:2).  A  thorough  understanding  of  the 
adversary  includes  the  lofty  goal  of  predicting  the  adversary’s  actions  before  they  put 
them  into  motion.  Limitation  of  this  knowledge  requires  the  commander’s  staff  to 
develop  feasible  courses  of  action  in  an  uncertain  environment. 
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Current  doctrine  and  planning  processes  strive  to  address  the  challenges  presented 
by  the  nature  of  this  leadership  task.  Joint  doctrine  provides  the  United  States  armed 
forces  overarching  guidance  to  achieve  the  President’s  and  Secretary  of  Defense’s 
strategic  goals  by  bridging  the  gap  between  policy  and  employment.  It  describes  war  as, 
“. .  .a  complex,  human  undertaking  that  does  not  respond  to  deterministic  rules”  (JP  1, 
2007:Sec.  I,  1).  An  adversary  in  war  plays  by  many  sets  of  rules  which  do  not  often 
remain  stagnant.  Neither  commanders  nor  their  staffs  fully  know  the  intentions  of  an 
adversary,  especially  in  today’s  information  age  and  in  the  face  of  asymmetric  warfare 
where  targets  and  tactics  change  in  a  matter  of  minutes.  This  produces  a  capability  gap 
between  a  commander’s  desire  and  capabilities  the  commander  possesses. 

Joint  Publication  3-0,  Joint  Operations,  defines  a  system  as  “a  functionally  related 
group  of  elements  forming  a  complex  whole”  (JP  3-0,  2006:Sec.  II,  21).  This  definition 
includes  the  complex  interaction  of  political,  military,  economic,  social,  information,  and 
infrastructure  (PMESII)  elements  in  which  today’s  military  must  operate.  The  doctrine 
stresses  this  system  perspective  in  order  for  planners  to  gain  a  better  understanding  of  the 
interactions  within  and  between  friendly,  adversarial,  and  neutral  systems  (JP  3-0, 
2006:Sec.  II,  22). 

In  addition  to  extensive  system  understanding,  joint  doctrine  asks  a  Joint  Force 
Commander  (JFC)  to  employ  unified  action.  Unified  action  consists  of  synergistically 
applying  all  instruments  of  national  and  multinational  power  (JP  3-0,  2006:Sec  II,  3). 
Instruments  of  national  power  include  diplomatic,  infonnation,  military,  economic, 
financial,  intelligence,  and  law  enforcement  (DIMEFIL)  actions  at  the  disposal  of  a 
nation  or  coalition  of  nations.  A  JFC  must  integrate  and  synchronize  the  actions  of  joint 


2 


US  military  forces,  multinational  forces,  intergovernmental  organizations, 
nongovernmental  organizations,  and  other  US  government  agencies  to  achieve  unified 
action  (JP  3-0,  2006:Sec.  II,  4). 

Commanders  desire  the  capability  to  continuously  assess  the  progress  towards 
achievement  of  their  objectives  in  today’s  environment  (JP  5-0,  2006:Sec  III,  57).  This 
capability  expands  a  commander’s  solution  space  for  selecting  a  feasible  course  of  action 
(COA).  A  commander  should  assess  the  direct  and  indirect  effects  of  a  COA  on  an 
adversary’s  political,  military,  economic,  social,  information,  and  infrastructure 
(PMESII)  systems  (JP  3-0,  2006:  Sec.  IV,  9).  COAs  consisting  of  purely  kinetic  or 
purely  military  options  may  limit  the  ability  to  best  achieve  a  commander’s  desired  effect 
used  to  reach  a  certain  end-state.  Instead,  one  should  evaluate  feasible  COAs  consisting 
of  all  elements  of  national  power  to  include  diplomatic,  infonnation,  military,  economic, 
financial,  intelligence,  and/or  law  enforcement  (DIMEFIL)  options. 

A  2004  Capabilities  Development  Document  (CDD)  for  Air  and  Space  Operations 
Center  (AOC)  as  a  Weapon  System  provided  guidance  for  developing  a  toolset  to  enable 
dynamic,  ongoing  effect  based  assessment  (EBA)  (CDD  AOC  WS,  2006:np).  Since  the 
Air  Force  Research  Laboratory  (AFRL)  has  the  mission  of  “leading  the  discovery, 
development,  and  integration  of  affordable  warfighting  technologies  for  America’s 
aerospace  forces”  (AFRL  PA,  2007:np),  they  are  supporting  this  EBA  requirement.  The 
Air  Force  Research  Laboratory/Information  Directorate  (AFRL/RI)  develops  systems, 
concepts,  and  technologies  to  enable  the  warfighter  in  today’s  challenging  infonnation 
age  (AFRL  PA,  2007:np).  A  current  focus  of  AFRL/RI  is  the  development  of  tools  to 
support  a  commander,  such  as  the  Joint  Force  Commander  (JFC)  or  Joint  Forces  Air 
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Component  Commander  (JFACC),  in  making  more  informed  decisions  when  selecting 
courses  of  action  (CO As).  Two  programs  currently  under  development  in  response  to 
this  guidance  include  the  Commander’s  Predictive  Environment  (CPE)  and  the  Dynamic 
Air  and  Space  Effects-base  Assessment  (DASEA). 

One  decision  analysis  approach  considered  to  address  the  challenge  of  developing 
COAs  in  a  rapidly  evolving  environment  is  value-focused  thinking  (VFT).  This 
technique  focuses  on  what  is  important  (values)  to  the  decision  maker  (DM)  and  elicits 
weighting  of  the  values.  The  solicited  values  form  a  weighted  value  hierarchy  to  assist  in 
the  decision  making  process.  This  process  attempts  to  aid  the  decision  maker  in  the  COA 
Development  portion  of  the  Joint  Operation  Planning  Process  (JOPP). 

Inappropriate  weights  caused  either  by  subjectivity  or  a  rapidly  changing  situation 
could  affect  the  decision.  Conducting  sensitivity  analysis,  by  varying  one  or  two  weights 
from  zero  to  one  and  changing  the  others  proportionally  to  see  how  the  decision  changes, 
attempts  to  determine  the  range  of  weights  for  which  the  model  recommends  the  same 
decision.  One-way  sensitivity  analysis  ignores  the  possible  interaction  between  two  or 
more  weights  in  a  hierarchy.  Two-way  sensitivity  analysis  addresses  the  interaction  of 
two  weights  in  the  hierarchy,  but  ignores  higher  interactions.  In  a  rapidly  changing 
environment,  such  as  combat,  are  the  solicited  weights  appropriate,  or  in  a  feasible  range 
where  the  decision  remains  the  same?  If  the  weights  are  not  in  a  feasible  range,  do  they 
affect  the  decision?  Do  interactions  of  more  than  three  weights  affect  the  COA 
recommended  to  the  commander?  This  provides  a  method  to  address  these  questions. 
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I.B.  Problem  Statement 


Commanders  and  their  staffs  face  difficult  situations  requiring  them  to  make 
decisions  under  conditions  of  uncertainty.  Many  quantitative  methods  elicit  preferences 
from  them  as  model  inputs.  Reaching  a  consensus  on  subjective  preferences,  often 
resulting  in  weights,  sometimes  proves  difficult  for  a  group  decision  (Ewing  et  al, 
2006:41).  While  many  traditional  decision  analysis  methods  analyze  the  sensitivity  of  a 
decision  to  changes  in  one  factor  or  the  interaction  of  two  factors,  the  interactions  of 
more  than  two  factors  could  affect  the  outcome.  The  use  of  a  decision  model  in  an 
uncertain  world  requires  it  to  be  flexible  and  robust  to  these  interactions. 

I.C.  Research  Scope 

This  thesis  uses  value-focused  thinking  to  generate  and  assess  courses  of  action  to 
achieve  a  commander’s  objectives.  Preferably,  the  leading  commander  or  his/her  staff 
should  provide  the  needed  inputs  to  create  the  decision  model.  Sometimes  the 
commander  or  his/her  staff  lack  the  time  needed  to  provide  the  inputs  required  from  a 
decision  maker.  The  research  presented  here  uses  a  notional  example  of  stability 
operations  in  Iraq  during  2007.  In  the  absence  of  direct  inputs  from  General  David 
Petraeus,  the  commander  of  Multi-National  Force-Iraq,  his  September  2007  report  to 
Congress  on  the  situation  in  Iraq  provides  the  foundation  for  the  case  study.  After 
developing  an  additive  value  model  based  on  the  report,  this  thesis  primarily  focuses  on 
quantifying  sensitivity  analysis  for  multiple  simultaneous  weight  changes  and  how  they 
affect  the  decision. 
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I.D.  Assumptions 

Key  national  and  military  leaders  involved  in  Iraq  are  unreachable  to  solicit  their 
values  and  objectives  for  Iraqi  stability.  Therefore,  General  David  Petraeus’s  September 
2007  report  to  Congress  and  subject  matter  expert  (SME)  opinion  provide  the  values  of 
the  model.  Additionally,  the  value  model  meets  the  desirable  properties  of  small  size, 
operability,  and  is  an  additive  value  model. 

I.E.  Thesis  Organization 

This  thesis  consists  of  five  chapters.  Chapter  2  reviews  literature  covering 
affinity  diagramming,  value-focused  thinking,  sensitivity  analysis,  and  the  measurement 
of  model  robustness.  Chapter  3  develops  a  value  model  using  affinity  diagramming  and 
proposes  a  robust  sensitivity  analysis  technique  using  five  measures.  Chapter  4  employs 
the  value  model  to  evaluate  and  analyze  CO  As.  Finally,  the  analyst  evaluates  the  CO  As’ 
robustness  to  the  weights  using  the  proposed  technique.  Chapter  5  presents  the  results  of 
the  study,  the  contributions  and  limitations  of  the  work,  and  possible  areas  of  future 
research. 
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II.  Literature  Review 


II.A.  Introduction 

This  chapter  addresses  the  past  research  conducted  in  the  areas  of  affinity 
diagramming,  value-focused  thinking,  various  analysis  techniques,  and  sensitivity 
analysis.  The  use  of  affinity  diagramming,  or  the  similar  K-J  method,  defines  the 
structure  of  the  hierarchy.  Value-focused  thinking  (VFT)  develops  the  value  model 
based  on  the  hierarchy.  Sensitivity  analysis  then  evaluates  the  robustness  of  the 
developed  value  model.  Following  a  brief  introduction  to  joint  planning,  the  chapter 
examines  the  details  of  a  few  decision  analysis  studies  grouped  by  steps  within  the  VFT 
process. 

Joint  Doctrine  5-0,  Joint  Operation  Planning,  defines  the  roles  of  a  Joint  Force 
Commander  (JFC)  and  his/her  staff.  The  JFC  serves  as  the  decision  maker  and  ensures 
the  execution  of  plans  for  military  operations.  The  role  of  the  JFC’s  staff  consists  of 
supporting  the  commander  in  understanding  complex  systems,  planning  COAs, 
recommending  COAs  to  the  commander,  ensuring  mission  execution,  and  mission 
assessment.  This  challenge  includes  “making  decisions  faster  and  better  than  a  thinking, 
adaptive  enemy  in  an  environment  of  uncertainty”  (JP  5-0,  2006:Sec.  Ill,  3). 

The  late  Colonel  John  Boyd,  USAF  Ret.,  emphasizes  the  importance  of  this 
concept  through  his  O-O-D-A  loop  in  his  series  of  briefings,  “Patterns  of  Conflict” 
(Coram,  2002:  334).  O-O-D-A  stands  for  Observe,  Orient,  Decide,  Act.  Boyd  explained 
that  the  concept  of  the  O-O-D-A  loop  applies  in  any  form  of  competition,  whether  war, 
business,  sports,  and  so  forth.  He  further  explained  that  a  commander’s  ability  to  cycle 
through  the  O-O-D-A  loop  faster  than  an  adversary’s  O-O-D-A  loop  enables  a 


7 


commander  to  know  and  counter  the  adversary’s  planned  actions.  This  causes  confusion 
and  disorientation  in  the  mind  of  the  adversary  due  to  outdated  or  irrelevant  information 
(Coram,  2002:  335). 

The  Joint  Operation  Planning  Process  (JOPP)  consists  of  seven  steps  ranging 
from  planning  initiation  to  plan  or  order  development.  The  seven  steps  include: 

Initiation,  Mission  Analysis,  COA  Development,  COA  Analysis  and  Wargaming,  COA 
Comparison,  COA  Approval,  and  Plan  or  Order  Development  (JP  5-0,  2006:Sec.  Ill,  20). 
COA  development  and  comparison  fall  within  the  scope  of  this  thesis  with  a  primary 
focus  on  a  new  paradigm  for  COA  comparison. 

II.B.  Affinity  Diagramming 

Affinity  diagramming  is  a  business  tool  used  to  organize  thoughts  and  ideas  into  a 
structured  hierarchy.  The  use  of  affinity  diagramming  for  organization  of  thought 
expands  beyond  the  business  world.  Analysts  use  the  technique  to  develop  decision 
analysis  models  based  on  documents  as  opposed  to  input  from  decision  makers  or  subject 
matter  experts. 

Parnell  et  al  employed  affinity  diagramming  in  the  Air  Force  2025  Study.  The 
team  established  Gen.  Fogleman’s  statement,  “achieve  air  and  space  dominance”  as  the 
objective  of  the  study  (Parnell  et  al,  1998: 1340).  The  inputs  received  from  40  teams 
resulted  in  109  action  verbs  (Parnell  et  al,  1998: 1340).  The  team  created  14  groups  of 
subtasks  and  labeled  each  group  with  one  of  the  verbs  from  within  the  group.  Next,  the 
team  identified  6  subtasks  as  tasks  and  grouped  the  remaining  8  subtasks  into  2  additional 
tasks  (Parnell  et  al,  1998:1341).  Finally,  the  team  grouped  the  8  tasks  under  3  categories 


they  called  functions  (Parnell  et  al,  1998:1344).  This  organization  process  developed  a 
hierarchical  structure  suitable  for  a  value-focused  thinking  approach. 

Pruitt  models  homeland  security  using  a  value  model  structured  through  affinity 
diagramming.  He  extracted  363  objectives  from  five  homeland  security  documents 
(Pruitt,  2003:3-15).  After  grouping  common  objectives,  Pruitt  defined  the  three  top 
objectives  as  Prevention,  Vulnerability  Reduction,  and  Response  Preparedness  (2003:3- 
15).  Sub -objectives  fell  accordingly  under  these  top  objectives. 

Affinity  grouping,  similar  to  affinity  diagramming,  groups  common  objectives  to 
form  a  hierarchical  structure.  Affinity  grouping,  however,  does  not  follow  the 
grammatical  rule  of  noun-verb  pairing.  Fensterer  uses  affinity  grouping  to  develop  a 
value  model  to  evaluate  stability  operations  (2006:53).  He  uses  the  25  tasks  found  within 
DoD  Directive  3000.05  to  establish  his  hierarchical  structure  into  five  main  objectives 
(Fensterer,  2006:56).  His  sub-objectives  come  from  the  364  tasks  identified  in  three 
publications  from  subject  matter  experts  (Fensterer,  2006:56-87,125-142). 

II.C.  Value-Focused  Thinking 

Ralph  Kenney  developed  value-focused  thinking  (VFT)  as  a  proactive  decision 
analysis  approach  as  opposed  to  a  reactive  one.  Kenney  offers  a  comparison  between 
what  he  calls  alternative-focused  thinking  (AFT)  and  VFT  (1992:47).  AFT,  as  a  reactive 
decision  analysis  technique  waits  for  one  or  more  alternatives  to  arise  and  then  selects  the 
best  one.  Value-focused  thinking  allows  a  decision  maker  (DM)  to  be  proactive.  A  DM 
considers  the  “things”  that  are  important  to  him/her.  These  “things”  are  values.  With  the 
values  defined,  a  DM  can  generate  alternatives  that  maximize  their  achievement.  When  a 
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decision  arises,  the  DM  knows  his/her  values  and  is  able  to  generate  a  list  of  alternatives 
that  are  at  least  as  good  as  those  generated  by  AFT. 

Value-focused  thinking  implements  a  ten  step  process  ranging  from  problem 
definition  to  providing  recommendations  to  a  DM  (Shoviak,  2001 :63).  The  first  five 
steps  result  in  the  development  of  the  value  model.  The  next  two  steps  deal  with  the 
alternatives.  The  final  three  steps  are  the  analysis  and  recommendations.  First,  define 
the  objective.  The  objective  provides  definition  to  the  problem.  Next,  develop  a  value 
hierarchy.  This  is  where  affinity  diagramming  can  come  into  play  to  organize  the  values 
into  a  logical  hierarchy.  Upon  completion  of  the  hierarchy,  develop  evaluation  measures 
for  the  values  in  the  lowest  tiers.  The  analyst  creates  value  functions  to  score  alternatives 
with  evaluation  measures.  Often,  these  functions  are  single  dimensional  value  functions 
(SDVFs).  Finally,  weighting  the  hierarchy  concludes  the  building  process  for  the  value 
model.  The  process  then  turns  its  attention  to  the  alternatives.  Once  the  value  model  is 
complete,  generate  alternatives  to  maximize  the  fulfillment  of  the  DM’s  objectives.  The 
alternative’s  fulfillment  of  each  objective  are  then  scored  using  the  value  functions.  The 
analysis  of  the  alternatives  then  begins.  Detenninistic  analysis  uses  the  hierarchy  weights 
and  scores  from  the  value  functions  to  find  each  alternative’s  overall  value  score.  The 
alternative  with  the  highest  score  is  preferred.  Sensitivity  analysis  varies  one  or  more 
weights  to  detennine  if  the  top  alternative  changes.  Finally,  the  analyst  presents  the 
results  of  the  study  and  provides  recommendations  to  the  DM.  Military  analysts  have 
used  this  methodology  for  course  of  action  (COA)  authoring  and  assessment.  The 
remainder  of  the  chapter  examines  some  of  those  studies. 
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II.D.  VFT  for  COA  Decisions 


In  2006,  Phipps  examined  the  use  of  VFT  for  COA  selection  from  a  Department 
of  Defense  perspective.  She  structured  her  assessment  structure  based  on  instruments  of 
national  power  (Phipps,  2006:29).  These  instruments  include  Diplomatic,  Information, 
Military,  Economic,  and  Social  (DIMES).  She  evaluates  a  diplomatic  threat,  a 
conventional  option,  special  operations  forces,  and  a  nuclear  option  as  alternatives  to 
achieve  a  commander’s  intent  in  a  notional  military  scenario  (Phipps,  2006:34,  36).  The 
approach  proved  its  worthiness  based  on  its  adaptability  and  low  strain  on  time  and 
resources  (Phipps,  2006:50-51). 

Fensterer  applied  VFT  to  the  planning  and  assessment  of  stability  operations  in 
2007.  He  developed  a  value  hierarchy  through  affinity  grouping  of  objectives  from 
doctrine  and  subject  matter  experts  (Fensterer,  2007:53-91).  The  value  hierarchy 
developed  reflects  strategic  level  objectives,  but  only  incorporates  notional  evaluation 
measures  (Fensterer,  2007:7,  91-97).  He  suggests  modeling  and  simulation  to  improve 
COA  outcome  prediction  accuracy  (Fensterer,  2007:1 16).  The  Stabilization  & 
Reconstruction  Operations  Model  (SROM)  developed  by  Robbins  provides  one  possible 
simulation  source.  The  model  assists  users  in  examining  the  interaction  of  factors 
governing  the  outcomes  of  stability  operations  (Robbins,  2005:7-8). 

While  the  theses  by  Phipps  and  Fensterer  demonstrated  the  notional  use  of  VFT 
for  COA  selection,  its  use  by  the  military  appears  frequently  in  the  literature.  The 
applications  range  from  Base  Realignment  and  Closure  (BRAC)  by  the  Army  and 
developing  future  air  and  space  forces  by  the  Air  Force  to  selecting  Information 
Operations  (IO)  COAs,  Psychological  Operations  (PSYOPS)  COAs,  and  automatic  target 
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recognition  systems  (Ewing  et  al,  2006:np;  Parnell  et  al,  1998:np;  Doyle  et  al,  2000:np; 
Kerchner  et  al,  2001  :np;  Bassham  et  al,  2006:np). 

The  applications  examined  here  all  use  the  same  basic  methodology,  but 
implement  it  differently  and  at  different  decision  levels.  Many  times,  the  analysts 
develop  at  least  an  initial  value  hierarchy  structure  based  on  documents  and  then 
supplement  them  with  decision  maker  and  subject  matter  expert  opinion.  Other 
applications  begin  with  the  decision  maker  and/or  subject  matter  experts.  The  weighting 
technique  often  varies  between  projects.  The  basic  VFT  process  ranks  the  alternatives, 
examines  the  sensitivity  of  the  decision,  and  then  presents  the  results.  The  applications 
examined  here,  however,  often  implement  additional  analysis  prior  to  the  sensitivity 
analysis. 

II.E.  Value  Hierarchy  Creation 

Value-focused  thinking  aims  to  collect  the  objectives  and  values  of  the  decision 
makers  to  support  achieving  the  objectives,  then  directly  evaluates  characteristics  of 
alternatives.  Studies  often  begin  when  senior  decision  makers  initiate  them. 

The  Air  Force  2025  Study  began  based  on  statements  from  then  Air  Force  Chief 
of  Staff,  General  Ronald  R.  Fogelman,  to  the  Air  University  (Parnell  et  al,  1998: 1336). 
The  study  team  initially  searched  for  the  gold  standard  objectives  to  develop  the 
hierarchy,  but  implemented  the  silver  standard  approach  (Parnell  et  al,  1998: 1336). 
Parnell  defines  gold  standard  as  a  model  developed  from  strategic  documents  and  silver 
standard  as  models  developed  based  on  data  from  the  stakeholder’s  representatives  when 
gold  standard  documents  are  not  sufficient  and  analysts  cannot  access  senior  decision 
makers  or  stakeholders  (2007:626).  The  team  examined  gold  standard  documents  such  as 
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the  National  Security  Strategy,  National  Military  Strategy,  Defense  Planning  Guidance, 
Joint  Vision  2010,  and  others,  but  none  met  all  the  study  criteria  (Parnell  et  al, 
1998:1339-1340).  Instead,  the  study  used  affinity  diagramming  to  organize  objectives 
identified  by  participants  (Parnell  et  al,  1998: 1340). 

In  the  development  of  10  COAs,  Doyle  searched  gold  standard  documents, 
technical  references,  and  used  the  opinions  of  subject  matter  experts  to  develop  a 
hierarchy  (2000:6).  Kerchner,  with  review  by  PSYOP  experts,  developed  a  value 
hierarchy  to  examine  the  psychological  operations  based  on  four  US  military  doctrinal 
publications  (2001:46-47).  The  2005  Army  BRAC  analysis  reviewed  several 
government  documents  relating  to  defense  transformation,  stationing,  and  BRAC  for 
objectives  and  supplemented  them  with  interviews  of  stakeholders  and  key  senior 
military  leaders  (Ewing  et  al,  2006:36). 

Sometimes,  decision  makers  and  stakeholders  are  accessible.  In  these  instances, 
analysts  directly  solicit  objectives  and  the  hierarchy  structure  from  the  decision  maker. 
Eareckson  Air  Station  in  Alaska  found  itself  not  meeting  regulations  for  municipal  solid 
waste.  Through  collaboration  with  the  decision  maker,  analysts  developed  a  value 
hierarchy  to  assist  the  decision  maker  in  developing  environmentally  compliant  strategies 
(Chambal  et  al,  2003:25,  27-28).  A  study  to  select  automatic  target  recognition  (ATR) 
systems  used  subject  matter  experts  to  develop  value  hierarchies.  A  single  expert  from 
the  Air  Force  Research  Laboratory  represented  the  ATR  evaluation  community  to 
develop  one  hierarchy  while  four  subject  matter  experts  from  Headquarters  Air  Combat 
Command  created  the  warfighter  hierarchy  (Bassham  et  al,  2006:52). 
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II.F.  Weighting 

After  completion  of  the  value  hierarchy  and  defining  of  evaluation  measures,  the 
analyst  solicits  weights  for  each  objective.  The  weights  indicate  the  decision  maker’s 
and/or  stakeholder’s  preferences.  The  studies  examined  here  used  several  different 
weighting  techniques. 

Air  Force  2025  aimed  to  identify  future  systems  needed  in  the  2025  timeframe. 
Analysts  did  not  know  the  future  state  of  the  world;  therefore,  they  developed  six 
possible  futures  in  which  to  evaluate  alternatives  (Parnell  et  al,  1998: 1346-1347).  The 
authors  do  not  discuss  the  exact  weighting  technique  used  in  this  application.  They  do 
note  that  each  of  forty  teams  submitted  weights  for  the  hierarchy  for  each  of  the  six 
alternate  futures.  The  weights  for  the  objectives  under  each  future  were  the  average 
weight  across  the  submissions  from  all  forty  teams  (Parnell  et  al,  1998:1347).  The  10 
study  does  not  discuss  the  weighting  method  used  either,  but  has  two  representatives  each 
weight  the  hierarchy  for  comparative  analysis  (Doyle  et  al,  2000:9-10). 

The  tradeoff  method  involves  soliciting  the  perceived  increase  in  value  when  a 
measure  moves  from  its  least  preferred  level  to  its  most  preferred  level  when  all  other 
measures  are  at  their  least  preferred  level  (Kerchner  et  al,  2001 :52).  The  analyst  then 
rank  orders  the  measures  based  on  value  increments  and  the  decision  maker  allocates 
percentages  to  the  objectives.  Kerchner  uses  this  method  to  evaluate  psychological 
operations  (2001:52-53). 

Another  weighting  technique  is  swing  weighting,  or  formally  known  as  Simple 
Multiattribute  Rating  Technique  using  Swings  (SMARTS)  (Ewing  et  al,  2006:41).  The 
ATR  system  evaluation  study  uses  this  method  for  both  the  evaluator  and  warfighter 
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frameworks  (Bassham  et  al,  2006:52,  54).  The  Army  BRAC  goes  beyond  SMARTS  to 
use  a  Swing  Weight  Matrix.  This  technique  places  all  measures  into  a  matrix  according 
to  importance  and  variation.  The  stakeholders  then  assign  swing  weights  to  all  measures. 
Global  weights  result  from  the  normalizing  of  all  swing  weights  in  the  matrix  (Ewing  et 
al,  2006:  41-42). 

II.G.  Additional  Analysis 

After  ranking  all  alternatives,  analysts  often  conduct  additional  analysis.  Value 
versus  cost  reveals  dominant  alternatives  and  provides  additional  insight.  The 
Information  Operations  study  charted  the  value  of  each  alternative  against  the  associated 
multiattribute  cost  value  (Doyle  et  al,  2000: 11-12).  The  cost  value  came  from  a  separate 
cost  hierarchy  (Doyle  et  al,  2000:7).  Analysts  used  this  same  methodology  to  evaluate 
psychological  operations  (Kerchner  et  al,  2001:60-61).  In  addition  to  evaluating  value 
versus  cost,  Doyle  et  al  solicited  most  likely  score  and  range  of  scores  for  each  attribute 
(2000: 10-1 1)  This  allowed  him  to  evaluate  COAs  while  taking  the  uncertainty  of  the 
measures  into  consideration.  Air  Force  2025  considered  this  type  of  analysis,  but  future 
technologies  do  not  have  cost  associated  with  them.  Instead  of  cost,  Air  Force  2025 
compares  value  to  the  challenge  to  develop  the  technology  needed  for  future  systems  to 
reach  maturity  (Parnell  et  al,  1998: 1348).  Doyle  created  characteristic  plots  for  the  top 
tier  objectives  in  the  hierarchy.  These  characteristic  plots  allowed  for  the  comparison  of 
value  for  each  objective  between  the  two  decision  makers  (Doyle  et  al,  2000: 12-13).  The 
Army  BRAC  study  wanted  to  optimize  the  selection  of  multiple  alternatives  subject  to  a 
budget  constraint,  but  they  did  not  have  the  needed  information.  Instead,  the  study  team 
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implemented  portfolio  analysis  to  determine  the  minimum  mix  of  alternatives  that  met 
the  Army’s  requirements  (Ewing  et  a/,  2006:42-46). 

II.H.  Sensitivity  Analysis 

According  to  Clemen,  sensitivity  analysis  answers,  “What  makes  a  difference  in 
this  decision?”  (2001 : 175).  Sensitivity  analysis  can  also  indicate  the  robustness  of  a 
model  under  the  uncertainty  in  the  weights.  A  model  is  most  robust  to  a  decision  if  the 
ranking  of  alternatives  does  not  change  when  weights  are  varied.  The  systematic 
approach  to  varying  the  weights  is  below. 

Sometimes,  an  analyst  knows  one  or  more  weights  with  certainty  and  prefers  to 
keep  them  constant.  Kahraman  develops  a  parametric  sensitivity  analysis  technique 
using  hierarchical  value  models.  The  technique  holds  one  or  more  weights  constant 
while  varying  the  others  proportionally  using  a  weight  coefficient  of  elasticity 
(Kahraman,  2002:32-38).  He  recommends  bounding  the  coefficients  of  elasticity  to 
maintain  the  assumptions  of  positive  weights  and  the  weights  summing  to  one. 

An  assessment  for  radioactive  disposal  reviews  five  sensitivity  techniques  before 
selecting  one  for  implementation  (Helton,  1993:327-328).  He  separates  these  techniques 
into  one  informal  and  four  fonnal.  The  informal  method  described  varies  one  parameter 
at  a  time  and  evaluates  the  changes  that  occur.  The  four  formal  approaches  include 
differential  analysis,  Monte  Carlo  analysis,  response  surface  methodology,  and  the 
Fourier  amplitude  sensitivity  test  (FAST)  (Helton,  1993:327). 

Differential  analysis  uses  a  Taylor  series  to  approximate  the  model  (Helton, 
1993:327).  The  initial  inputs  are  base  values,  ranges,  and  distributions  for  all  variables. 
A  vector  of  base  values  forms  the  starting  point  for  the  Taylor  series.  The  Taylor  series 
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approximates  the  function.  Variance  propagation  techniques  use  the  ranges  and 
distributions  to  estimate  the  uncertainty  in  the  approximated  function.  Finally,  the  Taylor 
series  estimates  the  significance  of  each  variable  (Helton,  1993:328). 

Monte  Carlo  analysis  results  from  multiple  probabilistic  model  runs.  Helton 
breaks  it  into  five  steps  (1993:330-331).  First,  select  a  range  and  distribution  for  each 
variable.  Second,  generate  a  sample  from  the  ranges  and  distributions  for  each  variable. 
Third,  evaluate  the  model  based  on  the  sample  generated  in  the  second  step.  Fourth, 
summarize  the  results  for  uncertainty  analysis,  often  as  a  mean  and  variance.  Finally, 
conduct  sensitivity  analysis,  either  through  scatterplots  or  stepwise  regression  analysis 
(Helton,  1993:330-331). 

Response  Surface  Methodology  (RSM)  creates  a  response  surface  that  serves  as  a 
surrogate  for  the  function.  Helton  breaks  RSM  into  6  steps  (1993:331-332).  Again,  the 
process  begins  by  defining  a  range  and  distribution  for  each  variable.  Second,  develop  an 
experimental  design  to  select  points  for  model  evaluation.  The  third  step  consists  of 
evaluating  each  point.  Fourth,  create  a  response  surface  based  on  the  results.  Fifth, 
expected  value  and  variance  or  Monte  Carlo  analysis  estimate  the  uncertainty  in  the 
function.  Finally,  evaluate  the  sensitivity  of  the  function  to  each  variable  by  evaluating 
the  importance  of  the  variables  in  relation  to  perturbations  of  the  variables  from  their 
expected  values  (Helton,  1993:331-332). 

The  FAST  approach  calculates  the  expected  value  and  variance  of  a  model 
prediction  through  a  four  step  process  (Helton,  1993:332).  First,  define  the  range  and 
distributions  for  each  variable  and  use  them  to  construct  a  density  function  for  the 
variable.  Next,  convert  the  multidimensional  integrals  for  expected  value  and  variance 
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from  the  first  step  into  one-dimensional  integrals.  Then,  estimate  the  expected  value  and 
variance  using  the  one-dimensional  integrals.  Finally,  a  variable’s  fractional  contribution 
to  variance  evaluates  the  sensitivity  of  the  function  (Helton,  1993:332-333). 

Bauer  et  al  compares  the  use  of  response  surface  methodology  (RSM)  against 
tornado  diagrams  and  strategy  region  graphs  for  sensitivity  analysis  of  influence 
diagrams  (1999:164).  Tornado  diagrams,  a  technique  for  one-way  sensitivity  analysis, 
show  a  rank  order  of  variability  for  each  factor.  Strategy  region  graphs,  a  two- 
dimensional  representation  of  two-way  sensitivity  analysis,  reveal  how  a  decision 
changes  based  on  the  interaction  of  two  factors.  He  states  the  benefits  of  using  RSM  for 
sensitivity  analysis  include  a  reduced  number  of  sensitivity  analysis  iterations,  an 
estimation  of  coefficients  at  which  a  decision  changes,  and  insight  into  the  interactions  of 
factors  (Bauer  et  al,  1999:162).  The  study  demonstrates  the  efficiency  of  RSM  and  the 
additional  insights  gained,  compared  to  other  techniques,  along  with  some  loss  of 
accuracy  resulting  from  approximations  by  the  RSM  equations  (Bauer  et  al,  1999:178- 
179).  While  not  explicitly  stated,  RSM  assumes  the  role  of  an  n-way  sensitivity  analysis 
technique. 

Bassham  et  al  implements,  in  an  ATR  selection  study,  three  sensitivity  analysis 
techniques  in  conjunction  with  the  value  hierarchy  (2006:58).  A  tornado  diagram  shows 
a  rank  order  of  the  variance  in  values  for  the  evaluation  measures.  Global  weights  for 
each  evaluation  measure  show  the  importance  of  each  measure,  but  Bassham  notes  it 
provides  little  insight  if  multiple  changes  occur.  Finally,  he  conducts  sensitivity  analysis 
using  saliency  measures.  Saliency  measures  show  the  contributions  of  the  inputs  with 
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respect  to  overall  value  score.  While  the  combat  model  evaluated  is  not  differentiable,  a 
response  surface  as  a  surrogate  for  the  model  is  (Bassham  et  al,  2006:58-62). 

n-way  sensitivity  analysis  is  an  extension  of  one-  and  two-way  sensitivity 
analysis.  Hughes  demonstrates  this  technique  to  assess  the  stability  of  a  completely 
subjective  decision  analysis  tree  (1990:68).  The  study  analyzes  the  stability  of  decisions 
made  by  surgical  nurses  in  various  situations  through  case  studies.  Through  the 
systematic  variation  of  weights,  analysts  determine  a  range  through  which  each  score  can 
vary.  If  these  ranges  do  not  overlap,  then  the  model  demonstrates  stability  (Hughes,  C. 
and  Hughes,  K.,  1990:68). 

Bednarski  applied  n-way  sensitivity  analysis  to  Bayesian  networks  and 
characterized  it  as  an  NP-hard  problem  (2003:29).  Due  to  the  difficulty  of  the  problem,  a 
genetic  algorithm  estimates  optimal  solutions,  but  cannot  guarantee  global  optima.  The 
study  shows  the  analyzed  network  insensitive,  but  investigators  hypothesize  that  this 
results  from  the  robustness  of  Bayesian  networks  (Bednarski  et  al.,  2003:  32). 

Rios  Insua  and  French  provide  a  framework  for  sensitivity  analysis  in  discrete 
multiobjective  decision  making.  They  propose  distance-based  tools  to  conduct  sensitivity 
analysis  (1991:180).  This  analysis  finds  regions  in  which  local  optima  may  lie  and 
eliminates  some  alternatives  due  to  dominance  (Rios  Insua  and  French,  1991:181). 

Another  approach  to  sensitivity  analysis  incorporates  mathematical  programming 
techniques.  These  techniques  often  evaluate  the  variation  required  in  initially  specified 
weights  to  cause  the  preferred  alternative  to  change.  These  models  look  for  the  new 
weighting  that  causes  a  specific  alternative  to  rank  above  all  others. 
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Barron  and  Schmidt  first  presented  a  least  squares  procedure  proposing  a 
quadratic  programming  approach  (1988: 123-124).  The  objective  function  minimizes  the 
sum  of  the  square  distance  between  the  initial  weight  and  a  new  weight.  The  initial  and 
new  weights  must  sum  to  one  and  all  weights  are  positive  (Barron  and  Schmidt, 
1988:123).  A  smaller  objective  function  value  indicates  greater  sensitivity. 

Wolters  and  Mareschal  propose  a  similar  approach  using  goal  programming. 

Each  weight  has  two  deviational  variables  indicating  an  increase  or  decrease  from  the 
initial  weight.  The  objective  function  tries  to  minimize  the  sum  of  these  deviational 
variables  (Wolters  and  Mareschal,  1995:284).  They  also  show  constraints  used  to 
preserve  relative  importance  of  weights,  penalize  deviational  variables  with  cost 
coefficients,  and  place  upper  and  lower  bounds  on  weights  (Wolters  and  Mareschal, 
1995:284-285). 

Ringuest  extends  the  work  of  Barron  and  Schmidt  and  Wolters  and  Mareschal  to  a 
generalized  Lp  metric.  Ringuest  titles  the  approach  of  Barron  &  Schmidt  the  Li  metric 
(1997:566-567).  Ringuest  strengthens  this  metric  by  combining  it  with  an  L»  metric. 

The  L«  metric  minimizes  the  maximum  deviation  of  a  single  weight.  The  deviations  of 
all  other  weights  remain  less  than  this  maximum  deviation  (Ringuest,  1997:567).  Using 
compromise  programming,  Ringuest  minimizes  both  his  Li  and  L»  metric  and  expresses 
it  as  the  Lp  metric  (Ringuest,  1997:566). 

This  thesis  focuses  on  the  examination  of  measures  defined  by  Barron  and 
Schmidt,  Wolters  and  Moreschal,  and  Ringuest’ s  Li  and  L»  metrics. 
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II.I.  Summary 

This  chapter  reviews  the  literature  in  the  fields  of  affinity  diagramming,  value- 
focused  thinking,  and  sensitivity  analysis.  Repeatedly,  affinity  diagramming  proves 
worthy  for  organizing  value-hierarchies  in  the  absence  of  an  overarching  decision  maker. 
While  VFT  is  a  complete  methodology  for  analysis;  analysts  often  link  the  method  with 
other  tools  like  portfolio  analysis.  A  key  point  to  each  study  using  VFT  is  the  sensitivity 
analysis.  Analysts  conduct  sensitivity  analysis  using  infonnal,  ad  hoc,  approaches  and 
more  formal  approaches  ranging  from  single  factor  to  multi-factor  analysis.  The  next 
chapter  builds  the  methodology  of  this  study,  demonstrating  the  use  of  VFT  to  author  and 
assess  course  of  action  and  a  robust  sensitivity  analysis  approach. 
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III.  Methodology 


III.A.  Introduction 

This  chapter  addresses  the  methods  used  to  develop  a  subjective  value  hierarchy 
and  evaluate  its  robustness.  Initially,  VFT  structures  a  commander’s  objectives  for 
course  of  action  selection  and  assessment  into  a  hierarchy.  The  technique  of  affinity 
diagramming  provides  a  starting  point  in  the  absence  of  a  specific  decision  maker.  This 
research  then  constructs  a  value  model  based  on  the  developed  hierarchy.  The  value 
model  provides  guidance  to  develop  CO  As.  Finally,  this  research  develops  sensitivity 
measures  for  all  alternatives  to  indicate  how  the  COA  is  sensitive  to  small  changes  in  the 
weights. 

III.B.  Extracting  Needed  Information 

Decision  analysis  typically  develops  decision  models  based  on  a  decision  maker’s 
inputs.  This  is  not  always  the  case  as  seen  in  the  Air  Force  2025  Study  (Parnell  et  al, 
1998: 1340).  When  analysts  cannot  reach  senior  decision  makers  due  to  location  or  time 
constraints,  they  can  base  models  on  the  available  information,  sometimes  in  the  form  of 
policy  documents  or  speeches  by  decision  makers.  These  sources  of  information  often 
contain  the  objectives  of  the  decision  maker,  measures  that  quantify  the  achievement  of 
objectives,  and  possible  actions  to  achieve  the  stated  objectives.  When  extracting  this 
information,  separating  it  into  these  three  categories  provides  organization  for  building 
the  model.  Capturing  this  infonnation  on  index  cards  or  similarly  sized  pieces  of  paper 
assists  in  the  later  organization  process  (Alloway,  1997:75-76). 

Stated  goals  and  objectives  in  the  source  should  reveal  what  a  decision  maker’s 
values  are  in  a  situation.  This  information  enables  the  analyst  to  begin  building  a  value 
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model.  Writing  down  each  goal  or  objective  from  the  source  on  a  separate  index  card 
and  including  the  source  from  which  it  came  provides  traceability  of  the  process.  Each 
card  should  follow  the  same  grammatical  format,  such  as  a  verb  followed  by  at  least  a 
noun.  This  format  lends  itself  to  the  affinity  diagramming  procedure  detailed  in  the  next 
section. 

Measures  indicate  an  achievement  of  goals  or  objectives.  Measures  included  in 
the  documents  or  speeches  provide  a  starting  point  by  showing  what  is  currently  used.  In 
addition  to  capturing  what  is  measured,  it  is  useful  to  capture  current  data  associated  with 
those  measures  as  well  as  the  desired  level  of  the  measure. 

The  final  piece  of  infonnation  to  capture  is  possible  actions.  These  actions 
provide  a  starting  point  to  develop  alternatives  for  the  decision  model.  An  alternative  can 
also  consist  of  multiple  actions.  After  capturing  all  the  needed  infonnation,  affinity 
diagramming  organizes  the  goals  and  objectives. 

This  research  uses  General  Petraeus’s  September  2007  Report  to  Congress  on  the 
Situation  in  Iraq  which  addresses  the  United  States  military’s  stability  objectives  in  Iraq, 
possible  actions,  and  measures  used  to  gauge  success.  Extracting  the  objectives  in  a 
verb-noun  phrase  supports  the  affinity  diagramming  process,  the  next  step  in  building  the 
decision  model.  The  overall  objective  described  by  the  report  is  a  stable  Iraq  (Petraeus, 
2007:2).  Table  3.1  shows  the  additional  infonnation  extracted  from  the  report. 
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Table  3.1  Extracted  Information  from  Petraeus  Report 


Objectives: 

-  Reduce  ethno-sectarian  violence/deaths 

-Grow  Iraqi  Security  Forces  &  shoulder 

-  Reduce  overall  civilian  deaths 

the  load 

-  Reduce  number  of  insurgent  crossing 
border 

-Establish  joint  security  stations 

Actions: 

-  Take  away  insurgent  sanctuaries  and 

-  Percent  Coalition  troops  support 

gain  the  initiative 

training  of  Indigenous  Security 

-  Disrupt  Shia  militia  extremists 

Institution  (Civil  Defense,  Indigenous 

—  capturing  the  head  and  other 

military,  Border  Patrol,  Facility 

leaders  of  Iranian  supported  special 

Protection) 

groups 

—  Number  of  training  classes 

—  neutralize  5  media  centers 

—  Number  of  facilities 

—  detained  senior  leaders 

-  Humanitarian  Relief 

—  captured  -100  other  key  leaders  & 
2500  rank-and-file  fighters 
-  Dialog  with  insurgent  groups  and  tribes 

-  Support  to  critical  infrastructure 

Measures: 

-  Tribal  rejection  of  Al-Qaeda 

-  Willingness  of  locals  to  serve  in  the 

-  No.  of  overall  civilian  deaths 

Army  and  Police  Service 

-  No.  of  ethno-sectarian  deaths 

—  140  Army,  Police  and  Special 

-  No.  of  attacks  (car  bombings,  suicide) 

Operations  Forces 

-  Coalition  losses 

—  95  capable  of  taking  the  lead 
-  445,000  individuals  on  payrolls  for 

Iraq’s  government 

While  the  information  extracted  does  not  provide  the  entire  model,  it  does  offer  a  starting 


point. 

III.C.  Affinity  Diagramming 

Affinity  diagramming  is  a  business  tool  used  to  organize  thoughts  and  ideas  into  a 
structured  hierarchy.  Based  on  a  sentence  that  summarizes  the  overall  problem, 
participants  gather  supporting  topics  from  documentation  with  descriptions  consisting  of 
a  noun-verb  pair  at  a  minimum  (Brassard  and  Ritter,  1994:13).  After  exhaustively 
gathering  support  topics,  grouping  similar  topics  together,  and  writing  a  sentence  to 
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summarize  the  topics  within  the  group  begins  the  organization  process.  Grouping  similar 
summarized  topics  and  again  summarizing  these  new  groups  begins  developing  the 
hierarchical  structure  needed  for  the  value  model.  Figure  3.1  provides  an  illustration  of 
the  hierarchical  structure.  Alloway  suggests  using  index  cards  for  the  process  for  easy 
reorganization  at  any  point  (1997:75-76). 


Overall 

Problem 


Figure  3.1  Hierarchical  Structure  from  Affinity  Diagramming 


Brassard  refers  to  the  K-J  method  as  a  variation  on  affinity  diagramming.  The 
key  differences  include  the  organization  of  fact-based  ideas  and  a  more  structured  process 
in  the  K-J  method  (Brassard  and  Ritter,  1994:16).  The  K-J  method,  invented  in  the  early 
1950’s  in  Japan  by  Jiro  Kawakita,  achieves  the  same  goal  as  affinity  diagramming 
(Scupin,  1997:234).  Kawakita  invented  the  method  when  trying  to  synthesize  large 
amounts  of  data.  Like  affinity  diagramming,  the  user  writes  topics  on  index  cards,  but 
then  shuffles  them.  Shuffling  index  cards  reduces  bias.  The  next  step  is  grouping  the 
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cards  by  topics,  similar  to  affinity  diagramming,  and  summarizing  the  groups.  Finally, 
drawing  the  hierarchy  based  on  the  groupings  provides  the  basic  structure  for  the  value 
model  (Scupin,  1997:236). 

Examining  the  objectives  from  Table  3.1  indicates  a  relationship  between  the  first 
three  objectives  and  another  relationship  between  the  last  two.  The  first  three  objectives 
seem  related  to  security  while  the  last  two  relate  to  supporting  transition  within  Iraq. 
Based  on  the  information  thus  far,  the  hierarchy  begins  taking  form  as  seen  in  Figure  3.2. 


Figure  3.2  Affinity  Diagram  based  on  Petraeus  Report 


The  affinity  diagram  provides  the  initial  framework  for  value  hierarchy  developed  in  the 
next  section. 

III.D.  Value  Hierarchy 

Affinity  diagramming  may  provide  initial  organization  and  structure  for  a  value 
hierarchy,  but  the  initial  hierarchy  may  lack  certain  desirable  properties.  The  value 
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hierarchy  includes  the  objectives  from  the  affinity  diagram,  refines  the  structure,  and  adds 
the  value  measures.  Kirkwood  lists  the  five  desirable  properties  of  a  value  hierarchy  in 
his  book  Strategic  Decision  Making  ( 1 997 : 1 6- 1 9).  These  properties  include: 
Completeness,  Nonredundancy,  Decomposability,  Operability,  and  Small  size. 

Completeness,  also  termed  collectively  exhaustive,  ensures  that  each  level  of  the 
hierarchy  includes  all  the  infonnation  needed  to  evaluate  the  next  higher  level 
(Kirkwood,  1997:16).  Nonredundancy,  or  mutual  exclusivity,  ensures  that  evaluation 
considerations  do  not  overlap  (Kirkwood,  1997:16-17).  Decomposability,  or  preferential 
independence,  ensures  that  a  change  in  one  evaluation  measure  does  not  cause  a  change 
in  another  evaluation  measure  (Kirkwood,  1997:17-18).  Preferential  independence 
allows  for  the  use  of  an  additive  value  function  for  the  value  model.  Operability  ensures 
that  the  value  hierarchy  is  understandable  and  usable  in  the  eyes  of  the  decision  maker 
(Kirkwood,  1997:18).  The  final  desired  property,  small  size,  emphasizes  the  point  that  it 
is  easier  to  explain  and  understand  a  smaller  hierarchy  as  opposed  to  a  larger  one 
(Kirkwood,  1997:18). 

The  evaluation  measures  show  the  level  of  achievement  for  the  lowest  level 
objectives.  Each  measure  has  a  combination  of  two  classifiers.  Measures  have  either  a 
natural  or  constructed  scale,  and  they  have  either  a  direct  or  proxy  scale  (Kirkwood, 
1997:24-25).  The  four  combinations  of  these  classifiers  rank  from  most  preferred, 
natural-direct,  to  least  preferred,  constructed-proxy.  Natural  measures  are  generally 
interpreted  the  same  by  everyone,  such  as  profit  in  dollars.  Analysts  develop  constructed 
scales  for  a  specific  decision  problem  and  explicitly  define  the  meaning  of  the  scale. 
Direct  scales  directly  measure  the  level  of  achievement  for  an  objective,  again  such  as 
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profit  in  dollars,  while  proxy  scales  show  a  level  of  achievement  for  an  objective,  but  do 
not  directly  measure  it,  such  as  gross  domestic  product  of  a  country  (Kirkwood,  1997:24). 

The  first  desirable  property  of  a  value  hierarchy  is  completeness.  Forming 
objectives  based  on  the  actions  and  measures  extracted  from  the  report  allows  for  a  more 
complete  hierarchy.  Adding  objectives  such  as  critical  infrastructure  and  humanitarian 
relief  makes  the  hierarchy  more  complete.  The  next  desirable  property  is  nonreduncancy. 
Overall  civilian  deaths  depends  on  the  deaths  associated  with  ethno-sectarian  violence, 
and  therefore  violates  the  nonredundancy  property.  Separating  civilian  deaths  into  those 
caused  by  ethno-sectarian  and  non-sectarian  violence  satisfies  this  property.  The  sub¬ 
objectives  under  Unit  Capability  Levels  in  the  hierarchy  below  present  problems  relating 
to  decomposability.  As  a  unit  becomes  more  or  less  proficient  in  their  training,  they 
move  from  one  capability  level  to  the  other.  A  change  in  the  measure  for  one  capability 
level  has  an  associated  change  in  the  measure  of  another  capability  level.  This  is  an 
example  of  preferential  dependence  in  the  model.  Simulation,  however,  shows  that 
consistent  and  reliable  results  are  achievable  as  long  as  the  value  model  is  close  to  being 
preferentially  independent  (Stewart,  1996:308).  Operability  and  small  size  are  subjective 
properties  with  their  satisfaction  determined  by  the  decision  maker.  The  assumption  is 
that  the  value  model  established  in  this  research  is  operable  and  small. 

After  modifying  the  hierarchy  to  meet  all  objectives  and  linking  measures  to  each 
objective,  the  hierarchy  takes  the  form  shown  in  Figure  3.3. 
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Figure  3.3  Value  Hierarchy  based  on  Petraeus  Report 


Appendix  A  contains  information  pertaining  to  the  evaluation  measures  including 
their  scales.  This  model  uses  fourteen  attributes  to  measure  a  course  of  action’s 
effectiveness  in  achieving  the  overall  objective,  a  stable  Iraq. 

III.E.  Single  Dimensional  Value  Functions 

Single  dimensional  value  functions  (SDVFs)  translate  evaluation  measures  to 
value  scores  ranging  between  zero  and  one.  They  may  be  continuous  or  discrete. 
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Continuous  forms  include  linear,  piecewise  linear,  exponential,  or  S  curve.  Discrete 
SDVFs  are  broken  into  bins.  Whether  SDVFs  are  increasing  or  decreasing,  they  must  be 
mono  tonic.  Graphs  of  these  value  functions  place  the  independent  evaluation  measure 
range  on  the  x-axis  with  the  value  ranging  between  zero  and  one  on  the  y-axis. 

Linear  value  functions  provide  proportional  translations  from  evaluation  measure 
to  value.  Increasing  linear  functions  follow  equation  3.1a  while  decreasing  linear 
functions  follow  equation  3.1b  shown  below: 


Hx) 


x  -Low 
High  -Low 


Equation  3.1a 


v(x)  ■ 


High  ~  Xj 
High  -Low 


Equation  3.1b 


Inserting  the  evaluation  measure  for  x;  provides  the  value  score  for  the  evaluation 
measure  (Kirkwood,  1997:65-66). 

Discrete  functions  divide  an  evaluation  measure  into  bins.  Categorical  measures 
lend  themselves  to  this  type  of  function.  The  least  preferred  category  receives  a  value  of 
zero  while  the  most  preferred  receives  a  value  of  one.  Assign  the  value  to  the  remaining 
categories  using  value  increments  (Kirkwood,  1997:  62).  For  example,  if  there  are  3 
categorical  measures,  A,  B,  and  C,  where  A  is  the  least  preferred  and  C  is  the  most 
preferred,  then  A  has  a  value  of  zero  and  C  has  a  value  of  one.  B  is  more  important  than 
A  by  a  value  increment  of  x.  C  is  more  important  than  B,  but  by  twice  the  increment 


between  A  and  B.  The  increment  between  A  and  B  is  x  and  the  increment  between  B  and 


C  is  2x,  then  these  increments  summed  together  are  3x  over  a  range  of  1 .  Solving  for  x,  1 
divided  by  3  results  in  x  equaling  0.33,  meaning  B  has  a  value  of  0.33. 
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Piecewise  linear  functions  allow  continuous  translation  over  categorical  measures 
with  different  rates  for  each  range.  The  SDVF  has  a  different  linear  equation  for  each 
range  of  the  evaluation  measure  and  corresponding  value.  The  method  of  value 
increments  determines  the  values  at  which  the  slope  of  the  piecewise  linear  function 
changes.  Increasing  functions  follow  equation  3.2a  for  all  ranges  while  decreasing 
functions  follow  equation  3.2b  shown  below: 

x.  —  Low 

v(x)  —  V  +  (-1  >  —  -I  >  ) - - -  Equation  3.2a 

V*/Vr  V  RangeLow  v  V  RangeHigh  V  RangeLoxv '  _ LOW 

High-  % 

V^X)  =  V Rangels  +  RangeHigh  ~  V RangeLoJ  High  _  Low  EqUati°n  ^ 

High/Low  and  RangeHigh/RangeLow  correspond  with  the  high  and  low  evaluation 
measure  and  value  for  each  piecewise  linear  section  of  the  overall  SDVF. 

Exponential  functions  can  estimate  a  piecewise  linear  function  over  the  range  of 
the  evaluation  measure  and  simplify  the  solicitation  process  since  it  requires  only  three 
data  points.  One  adjusts  the  curvature  of  the  function  by  changing  p  in  equation  3.3a  for 
increasing  exponential  functions  or  3.3b  for  decreasing  functions. 

-(jf-iow) 

1  —  p  p 

v(  -v- t  = - — -  Equation  3.3a 

^Xi>  -(High-Low) 

\~e  p 

-(High-x) 

1  —  0  P 

vt  v  1  = - — -  Equation  3.3b 

v^Xi>  -(High-Low) 

\-e  p 

Positive  values  of  p  cause  the  function  to  be  concave  while  negative  values  of  p  produce 
a  convex  function.  As  p  approaches  infinity,  the  function  approaches  linearity. 

Increasing  functions  with  a  negative  p  gives  more  value  to  evaluation  measures  higher  in 
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the  range;  positive  p  gives  more  value  to  lower  evaluation  measures.  Decreasing 
functions  work  in  the  opposite  manner  with  negative  p  giving  more  value  to  evaluation 
measures  closer  to  the  low  end  of  the  range  (Kirkwood,  1997:65-66).  The  S-curve  finds 
its  basis  in  the  exponential  function,  but  changes  convexity  at  a  point  within  the  range  of 
the  evaluation  measure  resulting  in  half  opening  concave  up  and  the  other  half  concave 
down.  Piecewise  linear,  exponential,  and  S-curve  SDVFs  allow  for  varying  returns  to 
scale  from  the  measure  to  the  associated  value. 

This  section  details  the  development  of  the  Estimated  Insurgent  Border  Crossing 
evaluation  measure  single  dimensional  value  functions  (SDVF).  Appendix  B  contains 
the  remainder  of  the  SDVFs.  Since  the  focus  of  this  research  is  the  approach  to  robust 
sensitivity  analysis,  the  author  does  not  detail  them  here.  The  evaluation  measures  range 
from  zero,  the  most  preferred  and  mapped  to  a  value  of  1,  to  one  hundred,  the  least 
preferred  and  mapped  to  a  value  of  0.  The  subject  matter  expert  (SME)  identified  that  10 
insurgents  crossing  the  border  per  month  achieves  a  value  of  0.5.  He  felt  that  10  or  more 
insurgents  entering  in  a  month  could  result  in  the  formation  of  a  new  terrorist  cell.  Given 
the  three  points  and  based  on  Equation  3.3b,  fitting  a  decreasing  exponential  curve  to  the 
data  points  results  in  a  p  of-14.4475,  shown  in  Figure  3.4. 
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Figure  3.4  Estimated  Insurgent  Border  Crossings  SDVF 

III.F.  Weighting  the  Hierarchy 

Assigning  weights  to  attributes  in  the  hierarchy  recognizes  relative  importance 
between  the  attributes.  The  sensitivity  analysis  techniques  described  in  this  research 
apply  to  global  weights  solicited  locally  using  swing  weights.  Local  weights  quantify  a 
decision  maker’s  preference  between  objectives  within  a  single  tier  of  the  hiearachy. 
Swing  weights  recognize  the  ranges  of  attributes  (von  Winterfeldt  and  Edwards, 
1986:275).  Solicitation  of  the  weights  involves  asking  a  decision  maker  to  first  rank, 
from  least  important  to  most  important,  the  evaluation  measure  range  for  all  attributes  in 
a  single  tier  of  the  hierarchy.  The  least  important  attribute  receives  the  value  of  k. 
Progressing  in  ascending  order  of  the  attributes,  the  decision  maker  offers  how  much 
more  important  it  is  to  swing  the  attribute  from  its  lowest  measure  to  its  highest  measure 
as  opposed  to  doing  the  same  with  the  least  important  attribute.  The  attribute  receives  the 
value  of  y k,  where  y  indicates  the  relative  importance  as  a  multiple  of  changes  in  the  least 
important  attribute.  After  soliciting  the  relative  importance  for  all  attributes  in  a  tier  of 
the  hierarchy,  divide  1  by  the  sum  of  the  yk’s.  This  provides  the  local  weight  for  the  least 
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important  attribute  in  the  tier.  The  remaining  attribute  weights  result  from  multiplying 
the  corresponding  v  by  the  value  of  k  as  seen  in  equation  3.4. 


Wry,k - 

Tv.k 


Equation  3.4 


After  calculating  the  local  weights  in  all  tiers  of  the  hierarchy,  one  can  then 
calculate  the  global  weights.  The  global  weights  are  the  product  of  the  local  weights 
starting  with  an  attribute  at  the  lowest  tier  and  multiplying  by  the  local  weight  from  the 
tier  above  it  until  reaching  the  top  of  the  hierarchy.  After  calculating  all  global  weights, 
they  sum  to  one. 

Starting  with  the  first  tier  of  the  hierarchy,  Security  and  Transition,  the  SME 
asserts  that  Transition  is  least  important  and  assigned  a  \k  while  Security  is  3.5  times  as 
important  as  Transition  and  assigned  a  3.5k.  The  two  sum  to  4.5  with  a  reciprocal  of 
0.222.  This  results  in  local  weights  of  0.222  for  Transition  and  0.778  for  Security.  Table 
3.2  shows  the  relative  importance  within  each  tier  of  the  hierarchy  along  with  the 
associated  local  weights. 


Table  3.2  Relative  Importance  and  Local  Weights 


Objective 

Relative 

Importance 

Local 

Weight 

Security 

3.5 

0.778 

Transition 

1 

0.222 

Objective 

Relative 

Importance 

Local 

Weight 

Coalition  Forces'  Safety 

2 

0.276 

Estimated  Insurgent  Border  Crossings 

1 

0.138 

Ethno-Sectarian  Violence 

2.5 

0.345 

Non-Sectarian  Violent  Deaths 

1.75 

0.241 
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Objective 

Relative 

Importance 

Local 

Weight 

Critical  Infrastructure 

1 

0.103 

Humanitarian  Relief 

6.25 

0.641 

Indigenous  Security  Institutions 

2.5 

0.256 

Objective 

Relative 

Importance 

Local 

Weight 

Electricity 

2 

0.333 

Fuel  (Heating/Cooking) 

1 

0.167 

Potable  Water 

3 

0.500 

Objective 

Relative 

Importance 

Local 

Weight 

Additional  People  Needed  for  ISI 

1 

0.143 

Locals'  Willingness  to  Serve 

2 

0.286 

Unit  Capability  Levels 

4 

0.571 

Objective 

Relative 

Importance 

Local 

Weight 

Capability  Level  I 

6 

0.500 

Capability  Level  II 

3 

0.250 

Capability  Level  III 

2 

0.167 

Capability  Level  IV 

1 

0.083 

The  sensitivity  analysis  methods  used  in  this  research  require  global  weights.  An 
attribute’s  global  weight  is  the  product  of  its  local  weight  and  the  attributes  beneath 
which  it  falls.  For  example,  Estimated  Insurgent  Border  Crossings  has  a  local  weight  of 
0.138  and  Security  has  a  local  weight  of  0.778,  so  Estimated  Insurgent  Border  Crossings 
has  a  global  weight  of  0.107.  Table  3.3  shows  the  global  weights  in  rank  order  for  the 
fourteen  attributes  that  have  an  evaluation  measure  attached  to  them.  Figure  3.5  shows 
the  global  weights  on  the  value  hierarchy. 
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Table  3.3  Global  Weights 


Objective 

Global 

Weight 

Ethno-Sectarian  Violence 

0.268 

Coalition  Forces'  Safety 

0.215 

Non-Sectarian  Violent  Deaths 

0.188 

Humanitarian  Relief 

0.142 

Estimated  Insurgent  Border  Crossings 

0.107 

Locals'  Willingness  to  Serve 

0.016 

Capability  Level  I 

0.016 

Potable  Water 

0.011 

Additional  People  Needed  for  ISI 

0.008 

Capability  Level  II 

0.008 

Electricity 

0.008 

Capability  Level  III 

0.005 

Fuel  (Heating/Cooking) 

0.004 

Capability  Level  IV 

0.003 
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Figure  3.5  Value  Flierarchy  with  Global  Weights 


III.G.  Course  of  Action  Development 

Courses  of  action  describe  the  proposed  approaches  to  achieving  the  decision 
maker’s  overall  objective.  Each  course  of  action  has  a  score  for  all  evaluation  measures. 
The  evaluation  scores  translate  into  values  using  the  corresponding  SDVF.  One  can  then 
calculate  the  value  score  for  the  alternatives.  Based  on  the  additive  value  function  seen  in 
equation  3.5,  the  value  for  an  alternative  is  the  sum  of  the  product  of  the  value  score  for 
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each  attribute  according  to  its  SDVF  and  its  corresponding  weight.  The  superscript  on  V 
differentiates  the  different  alternatives. 

k 

v1 = X  Wi  v  Of,)  Equati°n  3-5 

i= 1 

The  courses  of  action  are  then  ready  for  ranking  to  discover  the  preferred  order  of 
alternatives. 

The  notional  example  in  this  research  evaluates  ten  alternatives.  The  Status  Quo 
course  of  action  characterizes  the  situation  in  Iraq  in  September  2007.  The  other  nine 
alternative  courses  of  action  propose  different  approaches  that  may  improve  on  the 
achievement  of  the  overall  objective  compared  to  the  status  quo.  The  ten  proposed 
courses  of  action,  in  no  particular  order  include: 

•  Status  Quo 

•  Expel  A1  Qaeda-Iraq  (AQI) 

•  Train  Indigenous  Security  Institutions  (ISI) 

•  Institute  a  Military  Draft  in  Iraq 

•  Partial  Coalition  Withdrawal  from  Iraq 

•  Full  Coalition  Withdrawal  from  Iraq 

•  Pursue  Kurdistan  Workers  Party  (PKK)  terrorists  in  Northern  Iraq 

•  Establish  Self-Sustained  Agriculture  within  Iraq 

•  Establish  24  hour  Electricity  throughout  the  country 

•  Lower  Iraq’s  Unemployment  Rate 

Each  course  of  action  has  an  evaluation  score  for  each  of  the  fourteen  attributes 
shown  in  Table  3.3.  The  evaluation  scores  characterize  each  course  of  action  in  order  to 
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differentiate  between  them.  They  enable  the  value  hierarchy  to  evaluate  the  courses  of 
action.  Table  3.4  shows  the  notional  evaluation  measures  for  the  alternatives. 


Table  3.4 

1  COA  Evaluation  Measures 

% 

Coalition 

Lost 

# 

Insurgents 

Crossing 

Ethno- 

Sectarian 

Violence 

Death 

Rate 

Non- 

Sectarian 

Violent 

Death 

Rate 

Avg. 

hrs./day 

Status  Quo 

2 

50 

40 

35 

13 

Expel  AQI 

5 

20 

20 

30 

13 

Train  ISI 

1 

30 

20 

30 

15 

Institute  Draft 

4 

60 

50 

50 

11 

Partial 

Coalition 

Withdrawal 

1 

90 

40 

40 

12 

Full  Coalition 
Withdrawal 

0 

100 

50 

60 

8 

Pursue  PKK 

10 

80 

50 

70 

14 

Self-Sustained 

Agriculture 

2 

60 

25 

35 

13 

24  hr 

Electricity 

Established 

2 

40 

20 

35 

24 

Lower 

Unemployment 

2 

45 

25 

35 

13 

39 


L 

H20/person 

/day 

L 

Fuel/person/ 

day 

Tons  of 
Supplies 
Needed 

Estimated 
#  of 

Members 

Willingness 

Status  Quo 

47 

0.32 

270 

40000 

High 

Expel  AQI 

60 

0.50 

270 

20000 

High 

Train  ISI 

65 

0.45 

270 

20000 

High 

Institute  Draft 

45 

0.20 

270 

10000 

Low 

Partial  Coalition 
Withdrawal 

46 

0.30 

270 

60000 

Medium 

Full  Coalition 
Withdrawal 

45 

0.15 

270 

90000 

Low 

Pursue  PKK 

55 

0.25 

270 

50000 

Medium 

Self-Sustained 

Agriculture 

50 

0.37 

50 

35000 

High 

24  hr  Electricity 
Established 

80 

0.70 

150 

30000 

High 

Lower 

Unemployment 

70 

0.40 

100 

25000 

Medium 

#  of 
Units  at 
Level  I 

#  of 
Units  at 
Level  II 

#  of 
Units  at 
Level  III 

#  of 
Units  at 
Level  IV 

Status  Quo 

12 

83 

42 

23 

Expel  AQI 

15 

85 

40 

20 

Train  ISI 

50 

90 

15 

5 

Institute  Draft 

12 

83 

50 

30 

Partial 

Coalition 

Withdrawal 

12 

43 

21 

67 

Full  Coalition 
Withdrawal 

12 

10 

10 

118 

Pursue  PKK 

15 

85 

40 

20 

Self-Sustained 

Agriculture 

12 

83 

42 

23 

24  hr 

Electricity 

Established 

12 

83 

42 

23 

Lower 

Unemployment 

12 

83 

42 

23 
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III.H.  COA  Scoring  and  Ranking 

After  calculating  the  value  score  for  each  alternative,  ranking  them  in  descending 
order  provides  the  preferred  order  of  the  alternatives.  The  alternative  with  the  highest 
value  is  the  preferred  alternative.  At  this  point,  sensitivity  analysis  examines  the 
alternatives  and  how  changes  in  global  weights  may  change  the  preferred  alternative. 

This  analysis  shows  robustness  of  the  preferred  alternative  in  the  face  of  uncertainty  in 
the  weights. 

After  assessing  the  evaluation  scores  for  all  alternative  courses  of  action, 
quantitative  comparison  between  them  can  begin.  SDVFs  translate  each  evaluation  score 
to  an  associated  value  ranging  between  0  and  1.  A  course  of  action’s  value  score  is  the 
sum  product  of  the  fourteen  global  weights  in  Table  4.2  and  associated  values  derived  by 
the  SDVFs.  The  value  scores  also  range  between  0  and  1.  An  optimal  alternative  has  a 
value  score  of  1  because  it  scores  a  1  on  every  attribute.  A  rank  order  of  courses  of 
action  based  on  value  score  in  descending  order  shows  the  preferred  order  of  alternatives. 
TTT.T.  Traditional  Sensitivity  Analysis 

One-way  sensitivity  analysis  selects  one  factor  at  a  time  and  varies  the  weight 
from  zero  to  one.  All  other  weights  change  proportionally  to  maintain  the  relative 
weighting  between  all  factors.  Equation  3.6  shows  the  equation  for  finding  the  new 
weights  Wi  while  varying  Wk  between  zero  and  one,  where  k  indicates  the  weight  being 
varied  (Kirkwood,  1997:82,  84). 


i*k 


Equation  3.6 
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One-way  sensitivity  analysis  graphically  shows  a  change  in  decision  when  the 
value  lines  of  an  alternative  cross  in  two  dimensional  space.  The  top  line  in  the  graph 
shows  the  preferred  decision  over  the  specified  weight  range.  Kirkwood  explains  the 
process  involved  in  applying  one-way  sensitivity  analysis  to  a  value  hierarchy  (1997:82, 
84-85).  This  approach  requires  a  separate  chart  for  each  weight  in  the  model  which  can 
become  cumbersome  for  decision  problems  with  many  factors. 

Two-way  sensitivity  analysis  selects  two  factors  and  varies  both  of  them  from 
zero  to  one.  This  approach  examines  the  interaction  between  the  two  selected  factors. 

All  weights  must  still  sum  to  one.  Graphically,  two-way  sensitivity  is  three  dimensional. 
It  is  clear  that  the  graphical  representation  of  sensitivity  analysis  is  one  dimension  greater 
than  the  number  of  weights  varied.  This  limits  graphical  representation  of  sensitivity 
analysis  to  only  two  factors. 

III.J.  Robust  Sensitivity  Measures 

While  one-way  sensitivity  analysis  can  indicate  the  sensitivity  of  alternatives  to  a 
particular  weight,  it  does  not  take  multiple  non-proportional  weight  changes  into 
consideration.  Two-way  sensitivity  analysis  can  change  two  weights  in  a  non¬ 
proportional  manner  while  the  remaining  weights  change  proportionally  such  that  they  all 
continue  to  sum  to  one,  but  this  requires  the  problem  to  have  only  three  factors.  Since  all 
weights  can  vary  between  zero  and  one  with  the  only  constraint  that  they  must  sum  to 
one,  the  possibilities  of  weight  combinations  are  infinite. 

Small  changes  in  global  weights  indicate  high  decision  sensitivity.  Mathematical 
programming  can  minimize  a  defined  measure  measuring  the  changes  across  all  weights 
while  meeting  defined  constraints.  The  research  here  takes  this  approach  while  using 
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four  pro-defined  measures  and  presents  a  fifth.  Each  mathematical  program  minimizes 
the  measure,  while  ensuring  the  value  score  for  an  alternative  ranks  it  above  all  other 
alternatives,  the  new  set  of  weights  sums  to  one,  and  all  weights  are  non-negative.  This 
requires  solving  a  mathematical  program  for  each  alternative  for  each  measure.  If  a 
problem  is  infeasible,  then  the  alternative  cannot  rank  first  and  is  not  sensitive. 

Barron  and  Schmidt  first  presented  a  least  squares  measure  for  sensitivity  analysis 
of  multiattribute  value  models  (1988:123).  The  measure  is  the  sum  of  the  squared 
distance  between  the  original  weights,  W;  from  Table  3.3,  and  the  new  weights,  w,,  that 
minimize  this  measure.  The  quadratic  program  formulation  follows: 

k  2 

Problem  3.1a  Minimize  I  Or  i~Wi ) 

1=1 

subject  to  : 

Z  (v '  (JC,)  “  v"  (jt/))  Wi  ~  0  VA  *  B 

1=1 

iw,=i 

i=i 

iw=  i 

i=i 

o  -  Wi’W  i  - 1  v*  =  i»* 

where: 

Wj  =  the  initial  weights  defined  by  the  decision  maker 
W;  =  the  weights  found  that  minimize  the  measure 
vA(x;)  =  value  score  of  attribute  i  for  alternative  A 

The  problem  has  2k  +  n  +  1  constraints,  where  k  is  the  number  of  attributes  and  n  is  the 
number  of  alternatives.  Problem  3.1a  has  n  -  1  constraints  that  force  an  alternative  to 
rank  above  all  others.  There  are  constraints  requiring  the  initial  and  new  weight  scheme 
to  individually  sum  to  one.  Finally,  there  are  2k  boundary  constraints  requiring  the  initial 
and  new  weights  to  range  between  zero  and  one. 
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Wolters  and  Mareschal  asserted  a  linear  programming  approach  to  sensitivity 
analysis  of  additive  multiple  criteria  decision  making  methods  (1995:284).  The  measure 
is  the  sum  of  the  deviational  distances  between  the  original  weights  and  the  new  weights 
that  minimize  this  measure.  This  is  the  1-norm  of  the  vector  of  changes  in  weights.  The 
linear  program  formulation  follows: 

k 

Problem  3.1b  Minimize  X  CsC  +  s,) 

1=1 

subject  to  : 

£  (V*  <X>  -  V  (X,))  (W ,  +  s'  -  S ;>  ^  0  VA  *  B 

i= 1 

iw,‘  i 

i=l 

X  (5,+  -  S, )  = 0 

/= 1 

Sh  ’Si  -  0  V/  =  l..k 

0<W .  +  S*-S~<1  Vi  =  l..ifc 

where: 

Wj  =  the  initial  weights  defined  by  the  decision  maker 
Sj  =  the  positive  deviational  distance  from  Wf 
S;~  =  the  negative  deviational  distance  from  Wj 
vA(x;)  =  value  score  of  attribute  i  for  alternative  A 

This  problem  has  3k  +  n  +  1  constraints,  where  k  is  the  number  of  attributes  and  n  is  the 
number  of  alternatives.  There  are  n  -  1  constraints  requiring  an  alternative  to  rank  above 
all  others  by  changing  one  deviational  variable  or  the  other  for  each  weight.  The  initial 
weights  must  sum  to  one  and  the  sum  of  the  differences  between  deviational  weights  for 
each  attribute  must  sum  to  zero.  There  are  2k  constraints  requiring  the  non-negativity  of 
deviational  variables  and  an  additional  k  constraints  bounding  the  weights  between  zero 
and  one. 
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Ringuest  extended  Wolters  and  Mareschal’s  linear  programming  approach  by 
adding  the  infinity  nonn  (1997:566-567).  The  measure  minimizes  the  maximum  positive 
or  negative  deviation  from  initial  weights.  The  linear  program  formulation  follows: 

Problem  3.1c  Minimize  D 

subject  to : 

X (v J (JC/) ~ v“ (jc,)) (W ,  +  S*  ~ Si')-0 
g+,+g.<D  \/i  =  l..k 

tw,=' 

S%Si~°  Vi  =  )-k 

°-Wi+st~sirl  yi  =  l-k 
where: 

D  =  the  maximum  deviation  from  an  initial  weight  W; 

Wj  =  the  initial  weights  defined  by  the  decision  maker 
sfi  =  the  positive  deviational  distance  from  W; 

S;"  =  the  negative  deviational  distance  from  Wj 
vA(x;)  =  value  score  of  attribute  i  for  alternative  A 


Problem  3.1c  has  4k  +  n  +  1  constraints,  where  k  is  the  number  of  attributes  and  n  is  the 
number  of  alternatives.  There  are  n  -  1  constraints  requiring  an  alternative  to  rank  above 
all  others  by  changing  one  deviational  variable  or  the  other  for  each  weight,  k  constraints 
ensure  that  the  maximum  deviation  of  any  weight  is  at  least  as  small  as  the  objective 
function  value.  The  initial  weights  must  sum  to  one  and  the  sum  of  the  differences 
between  deviational  weights  for  each  attribute  must  sum  to  zero.  There  are  2k  constraints 
requiring  the  non-negativity  of  deviational  variables  and  an  additional  k  constraints 
bounding  the  weights  between  zero  and  one. 

Bauer  proposed  the  use  of  the  2-nonn  as  a  measure  for  sensitivity  analysis 
(2008:np).  Similar  to  Baron  and  Schmidt’s  least  squares  method,  the  2-norm  minimizes 
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the  square  root  of  the  sum  of  the  squared  distance  between  the  original  weights  and  the 
new  weights  that  minimize  this  measure.  The  quadratic  program  formulation  follows: 


Problem  3. Id  Minimize  JZ  (Wi  ~Wi ) 

subject  to  : 

Z(rAxI)-vii(xI))w,^0  VA*B 

1=1 

k 

Str,=i 

1=1 

k 

Ew,=1 

i=i 

0<W(<1  Vi  =  l.X 
where: 

Wj  =  the  initial  weights  defined  by  the  decision  maker 
W;  =  the  weights  found  that  minimize  the  measure 
vA(x;)  =  value  score  of  attribute  i  for  alternative  A 

This  problem,  like  the  least  squares  problem,  has  2k  +  n  +  1  constraints,  where  k  is  the 
number  of  attributes  and  n  is  the  number  of  alternatives.  Problem  3.  Id  has  n  -  1 
constraints  that  force  an  alternative  to  rank  above  all  others.  There  are  constraints 
requiring  the  initial  and  new  weight  scheme  to  individually  sum  to  one.  Finally,  there  are 
2k  boundary  constraints  requiring  the  initial  and  new  weights  to  range  between  zero  and 
one. 

The  infinity  norm  demonstrated  by  Wolters  and  Mareschal  captures  one  important 
piece  concerning  a  change  in  weight  in  terms  of  sensitivity.  When  global  weights  are 
small,  such  as  0.01,  a  change  to  0.02  is  a  change  of  0.01.  This  is  however,  a  200% 
change  in  weight.  The  new  proposed  approach  minimizes  the  maximum  percent  change 
across  all  weights.  The  linear  program  formulation  follows: 
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Problem  3 . 1  e  Minimize  P 
subject  to : 

Tj (V  (jc<) _ v 8 (jc,-))  ( +S*~Si'>-° 

(s‘  +Sj)  <P  V/  =  \..k 

w, 

iWi=i 

i<s:-s?= ° 

S  ’  s  —  ®  Vi  =  1..& 

0<fjf.  +  s+.-s~<\  Vi=lJfc 

where: 

P  =  the  maximum  percent  change  of  an  initial  weight  Wj 
W;  =  the  initial  weights  defined  by  the  decision  maker 
S;+  =  the  positive  deviational  distance  from  Wj 
Si"  =  the  negative  deviational  distance  from  Wj 
vA(Xj)  =  value  score  of  attribute  i  for  alternative  A 

Problem  3.  le  has  4k  +  n  +  1  constraints,  where  k  is  the  number  of  attributes  and  n  is  the 
number  of  alternatives.  There  are  n  -  1  constraints  requiring  an  alternative  to  rank  above 
all  others  by  changing  one  deviational  variable  or  the  other  for  each  weight.  The  k 
constraints  ensure  that  the  maximum  percent  change  of  any  weight  is  at  least  as  small  as 
the  objective  function  value.  The  initial  weights  must  sum  to  one  and  the  sum  of  the 
differences  between  deviational  weights  for  each  attribute  must  sum  to  zero.  There  are 
2k  constraints  requiring  the  non-negativity  of  deviational  variables  and  an  additional  k 
constraints  bounding  the  weights  between  zero  and  one. 

This  research  uses  Frontline  System’s  Premium  Solver  for  Education  V6.0  add-in 
with  Excel  2003.  The  software  has  three  engines  for  solving  mathematical  programs. 
This  research  used  the  Standard  Simplex  LP  engine  for  solving  the  linear  program 
associated  with  the  Problem  3.1b.  Solving  the  quadratic  programs  for  Problems  3.1a  and 
3. id  required  the  Standard  GRG  Nonlinear  engine.  This  engine  uses  a  generalized 
reduced  gradient  (GRG)  method  to  solve  the  quadratic  programs.  Finally,  the 
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mathematical  programs  with  non-smooth  objective  functions,  Problems  3.1c  and  3.1e, 
required  the  use  of  the  Standard  Evolutionary  engine.  This  third  engine  is  a  heuristic 
approach  that  estimates  the  optimal  solution  to  the  mathematical  program  and  it  cannot 
guarantee  its  optimality.  The  heuristic  approach  is  a  combination  of  both  genetic  and 
evolutionary  algorithms  along  with  classical  optimization  methods  (Premium  Solver  V8, 
2007:26).  While  this  section  introduced  the  measures  used  and  their  calculations,  the 
next  section  details  the  sensitivity  analysis  using  these  measures. 

III.K.  Robust  Sensitivity  Analysis 

The  five  measures  detailed  in  the  previous  section  provide  a  way  to  measure  the 
sensitivity  of  alternatives  to  small  changes  in  multiple  weights.  Smaller  measure  values 
indicate  greater  sensitivity.  The  top  alternative  according  to  the  value  model  possesses 
measure  scores  of  zero  across  all  measures.  Intuition  confirms  this  result  since  the 
current  weight  scheme  does  not  require  any  changes  to  rank  the  alternative  at  the  top. 
Alternatives  without  measure  values  due  to  infeasibility  in  the  mathematical  programs 
imply  an  insensitive  alternative.  The  insensitive  alternative  can  never  rank  on  top 
regardless  of  the  attribute  weighting  scheme. 

A  three-dimensional  bar  graph  representing  the  measure  values  across  all 
alternatives  and  measures  reveals  the  sensitive  alternatives.  For  the  proposed  alternatives 
in  this  research,  the  graph  will  indicate  a  clear  break  in  relative  measure  values  across  the 
alternatives,  thereby  separating  the  alternatives  into  two  groups,  sensitive  and  not 
sensitive.  Alternatives  falling  into  the  sensitive  group  across  all  five  measures  indicate 
sensitivity  and  require  further  examination.  Additional  resources  applied  to  these 
sensitive  alternatives  will  provide  a  refined  perspective  to  the  decision  maker. 
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III.L.  Summary 

This  chapter  presented  the  methodology  used  in  this  research.  Affinity 
diagramming  organizes  objectives  into  a  hierarchical  structure  in  the  absence  of  a 
decision  maker.  The  hierarchy  serves  as  the  basis  for  the  additive  value  model.  Once 
fully  developed,  alternatives  are  developed,  scored,  and  ranked.  Robust  sensitivity 
analysis  then  reveals  alternatives  sensitive  to  the  initial  weighting  scheme.  The  next 
chapter  applies  this  methodology  to  the  notional  stability  operations  problem  based  on 
General  Petraeus’s  September  2007  report  to  Congress. 
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IV.  Results  and  Analysis 


IV.A.  Introduction 

This  chapter  presents  the  results  of  the  research  conducted  using  the  methodology 
described  in  the  previous  chapter.  After  developing  the  ranking  of  the  alternatives,  one¬ 
way  sensitivity  analysis  shows  a  relatively  insensitive  value  model.  The  use  of  five 
robust  sensitivity  measures,  however,  indicates  alternatives  are  in  fact  sensitive  to 
simultaneous  changes  in  the  weights. 

IV.B.  COA  Scoring 

After  developing  the  value  model  and  constructing  alternative  courses  of  action 
with  their  respective  evaluation  measures,  the  analysis  begins.  The  first  step  uses  the 
SDVFs  to  translate  raw  evaluation  measures  to  values.  The  following  example  shows 
this  translation  for  the  #  Insurgents  Crossing  measure  of  the  Status  Quo  alternative.  An 
estimated  50  insurgents  cross  the  border  and  enter  Iraq  every  month  in  the  Status  Quo 
alternative.  This  measure  has  a  most  preferred  level  of  0  and  a  least  preferred  level  of 
100.  A  decreasing  exponential  curve,  with  a  p  value  of -14.4475  defines  the  SDVF. 
Based  on  Equation  3.3b,  the  Status  Quo  alternative  has  a  raw  score  of  50  and  a  value  for 
#  Insurgents  Crossing  of  0.03045  as  seen  in  Equation  4.1. 

-(100-50) 

\  —  „  -14.4475 

v(50)  = - -doo-50)  =  0.03045  Equation  4.1 

1  —  g  -14.4475 

Translating  the  remaining  evaluation  measures  to  value  for  each  course  of  action 
occurs  in  a  similar  manner  using  the  respective  SDVFs.  Table  4.1  shows  the  translated 
notional  values  for  all  alternatives. 
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Table  4. 1  CO  A  Evaluation  Values 


% 

Coalition 

Lost 

# 

Insurgents 

Crossing 

Ethno- 

Sectarian 

Violence 

Death 

Rate 

Non- 

Sectarian 

Violent 

Death 

Rate 

Avg. 

hrs./day 

Status  Quo 

0.990 

0.030 

0.112 

0.877 

0.852 

Expel  AQI 

0.906 

0.250 

0.566 

0.980 

0.852 

Train  ISI 

0.999 

0.125 

0.566 

0.980 

0.951 

Institute  Draft 

0.953 

0.015 

0.000 

0.038 

0.500 

Partial  Coalition 

Withdrawal 

0.997 

0.001 

0.112 

0.326 

0.720 

Full  Coalition  Withdrawal 

1.000 

0.000 

0.000 

0.004 

0.155 

Pursue  PKK 

0.077 

0.003 

0.000 

0.000 

0.913 

Self-Sustained  Agriculture 

0.990 

0.015 

0.412 

0.877 

0.852 

24  hr  Electricity 

Established 

0.990 

0.062 

0.566 

0.877 

1.000 

Lower  Unemployment 

0.990 

0.043 

0.412 

0.877 

0.852 

L 

H20/person/day 

L 

Fuel/person/day 

Tons  of 
Supplies 
Needed 

Estimated 
#  of 

Members 

Willingness 

Status  Quo 

0.001 

0.320 

0.000 

0.950 

1.000 

Expel  AQI 

0.132 

0.500 

0.000 

0.986 

1.000 

Train  ISI 

0.634 

0.450 

0.000 

0.986 

1.000 

Institute  Draft 

0.000 

0.200 

0.000 

0.995 

0.000 

Partial 

Coalition 

Withdrawal 

0.000 

0.300 

0.000 

0.857 

0.500 

Full  Coalition 
Withdrawal 

0.000 

0.150 

0.000 

0.380 

0.000 

Pursue  PKK 

0.024 

0.250 

0.000 

0.914 

0.500 

Self-Sustained 

Agriculture 

0.004 

0.370 

0.970 

0.962 

1.000 

24  hr 

Electricity 

Established 

0.997 

0.700 

0.128 

0.972 

1.000 

Lower 

Unemployment 

0.924 

0.400 

0.500 

0.980 

0.500 
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#  of  Units 
at  Level  I 

#  of  Units 
at  Level  II 

#  of  Units 
at  Level  III 

#  of  Units 
at  Level  IV 

Status  Quo 

0.060 

0.585 

0.338 

0.513 

Expel  AQI 

0.075 

0.575 

0.356 

0.560 

Train  ISI 

0.250 

0.550 

0.680 

0.865 

Institute  Draft 

0.060 

0.585 

0.275 

0.419 

Partial  Coalition 

Withdrawal 

0.060 

0.785 

0.582 

0.142 

Full  Coalition  Withdrawal 

0.060 

0.950 

0.773 

0.030 

Pursue  PKK 

0.075 

0.575 

0.356 

0.560 

Self-Sustained  Agriculture 

0.060 

0.585 

0.338 

0.513 

24  hr  Electricity 

Established 

0.060 

0.585 

0.338 

0.513 

Lower  Unemployment 

0.060 

0.585 

0.338 

0.513 

This  research  then  proceeds  to  find  an  overall  value  score  for  each  COA.  Based 
on  the  additive  value  function  shown  in  Equation  3.5,  the  Status  Quo  COA  has  a  value 
score  of  0.45 1  on  a  scale  from  0  to  1  as  seen  in  Equation  4.2  below. 

ySta,usQuo  =  .215(.990)  +  .1070030)  +  .2680 1 12)  + .  1880877) 

+  .0080852)  +  .01 10001)  +  .0040320)  +  .142(0)  +  ,008(.950)  +  .016(1)  Equation  4.2 
+  .0160060)  +  .008(.585)  +  ,003(.338)  +  ,003(.338)  +  ,003(.5 13)  =  .45 1 

The  value  score  then  shows  relative  comparisons  to  the  other  alternatives.  Table 

4.2  shows  the  calculated  value  score  for  each  COA  and  the  next  section  discusses  their 

rank  order. 
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Table  4.2  COA  Value  Scores 


Value  Score 

Self-Sustained  Agriculture 

0.668 

Train  ISI 

0.618 

24  hr  Electricity  Established 

0.608 

Lower  Unemployment 

0.607 

Expel  AQI 

0.600 

Status  Quo 

0.451 

Partial  Coalition  Withdrawal 

0.338 

Institute  Draft 

0.234 

Full  Coalition  Withdrawal 

0.214 

Pursue  PKK 

0.050 

IV.C.  COA  Ranking 

Ranking  the  evaluated  courses  of  action  in  descending  order  based  on  value 
provides  an  ordered  list  of  COAs  from  most  preferred  to  least  preferred.  An  optimal 
course  of  action  would  have  a  value  of  1  for  each  attribute  and  result  in  a  value  score  of 
1 .  The  Ideal  COA  shown  in  Figure  4. 1  provides  a  visual  reference  for  comparison.  As 
indicated  by  Figure  4.1,  the  Self-Sustained  Agriculture  alternative  is  the  recommended 
course  of  action  with  a  value  score  of  0.668.  The  next  four  alternatives,  Train  ISI,  24  hr 
Electricity  Established,  Lower  Unemployment,  and  Expel  AQI  all  score  relatively  close  in 
value  and  additionally  are  not  far  from  Self-Sustained  Agriculture.  In  an  uncertain 
environment,  it  is  possible  that  if  weights  or  evaluation  measures  are  imprecise,  then  a 
different  course  of  action  may  be  the  preferred  alternative.  This  is  where  sensitivity 
analysis  begins  and  is  the  focus  of  the  remainder  of  this  research. 
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Ranking  for  Stable  Iraq  (Sept.  '07)  Goal 


Alternative 

Value 

Ideal 

1.000 

Self-Sustained  Agriculture 

0.668 

Train  ISI 

0.618 

24  hr  Electricity  Established 

0.608 

Lower  Unemployment 

0.607 

Expel  AQI 

0.600 

Status  Quo 

0.451 

Partial  Coalition  Withdrawal 

0.338 

Institute  Draft 

0.234 

Full  Coalition  Withdrawal 

0.233 

Pursue  PKK 

0.050 

■ 

Ethno-Sectarian  Violence  Death  Rate 

■  %  Coalition  Lost 

■ 

Non-Sectarian  Violent  Death  Rate 

■ 

Tons  of  Supplies  Needed 

B  #  Insurgents  Crossing 

EJ 

Willingness 

□ 

#  of  Units  at  Level  I 

■  L  H20/person/day 

■ 

Estimated  #  of  Members 

■ 

#  of  Units  at  Level  II 

□  Avg.  hrs./day 

■ 

#  of  Units  at  Level  III 

■ 

L  Fuel/person/day 

■  #  of  Units  at  Level  IV 

Figure  4.1  Courses  of  Action  Rank  Order 
IV.D.  Traditional  Sensitivity  Analysis 

One-way  sensitivity  analysis  can  indicate  the  alternatives  sensitive  to  small 
changes  in  a  single  weight.  For  the  purposes  of  this  research,  a  change  in  weight  less 
than  0.05  resulting  in  a  different  alternative  being  preferred  indicates  sensitive 
alternatives  and  the  influence  weights.  In  the  problem  addressed  by  this  research,  a 
change  of  0.05  in  a  single  weight  is  a  20%  change  in  weight  if  considering  Ethno- 
Sectarian  Violence  Death  Rate,  the  attribute  with  the  highest  weight.  This  same  change 
of  0.05,  if  looking  at  Tons  of  Supplies  Delivered,  is  a  1250%  change  in  weight.  Most  of 
the  attributes  have  weights  similar  to  that  of  Tons  of  Supplies  Delivered.  While  a 
decision  maker  may  provide  imprecise  weights  for  attributes,  they  should  not  be  off  by 
more  than  20%.  One-way  sensitivity  analysis,  however,  can  only  show  this  analysis  one 
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attribute  weight  at  a  time.  For  example,  Figure  4.2  shows  the  sensitivity  graph  for  the 
measure  associated  with  Estimated  Insurgent  Border  Crossings.  The  weight  associated 
with  this  measure  would  have  to  increase  from  0. 107  to  greater  than  0.3  in  order  for 
Expel  AQI  to  become  the  preferred  alternative.  This  graph  does  not  indicate  one-way 
sensitivity  in  the  model. 


Self-Sustained  Agriculture 
Train  ISI 

24  hr  Electricity  Established 
Lower  Unemployment 
Expel  AQI 
Status  Quo 

Partial  Coalition  Withdrawal 
Institute  Draft 
Full  Coalition  Withdrawal 
Pursue  PKK 


Figure  4.2  Estimated  Insurgent  Border  Crossings  Sensitivity  Graph 
Further  examination  shows  that  all  attributes  require  a  change  in  weight  of  0.05  or 
greater  to  cause  the  preferred  alternative  to  change.  The  changes  needed  in  the  Potable 
Water  and  Humanitarian  Relief  weights  to  change  the  preferred  alternative  are  the  closest 
to  0.05  as  seen  by  Figures  C.7  and  C.8  in  Appendix  C,  but  are  greater  than  0.05.  Potable 
Water  has  a  weight  of  0.01 1,  a  change  of  0.05  is  at  455%  change;  while  Humanitarian 
Relief’s  weight  of  0.142  changing  by  0.05  is  a  35%  change.  Therefore,  these  attributes 
do  not  indicate  weight  sensitivity  in  the  model.  This  means  that  the  decision  model  is  not 
weight  sensitive  based  on  traditional  sensitivity  analysis  techniques.  Appendix  C  shows 
the  remainder  of  the  one-way  sensitivity  graphs.  While  a  change  in  one  attribute  weight 
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at  a  time  does  not  indicate  sensitivity,  it  is  possible  that  multiple  small  changes  in 
attribute  weights  could  indicate  sensitivity. 

IV.E.  Robust  Sensitivity  Measures 

Systematically  changing  all  weights  at  the  same  time  presents  a  challenge  due  to 
the  infinite  number  of  possible  weight  combinations.  The  sensitivity  measures  discussed 
in  the  previous  chapter  provide  structure  to  finding  different  combinations  of  weights  that 
could  indicate  sensitivity  or  insensitivity  in  a  value  model.  The  intent  of  all  the 
sensitivity  measures  is  the  same;  they  indicate  the  distance  between  the  assessed  weights 
from  Table  3.3  and  a  new  set  of  weights  that  causes  a  course  of  action  to  become  the 
preferred  alternative,  regardless  of  its  original  rank.  While  one-way  sensitivity  analysis 
examines  sensitivity  one  weight  at  a  time,  the  sensitivity  measures  used  here  examine  one 
alternative  at  a  time.  The  top-ranked  alternative  satisfies  all  constraints  used  to  calculate 
the  new  weights.  Therefore,  the  top  ranked  alternative,  Self-Sustained  Agriculture,  has  a 
sensitivity  measure  of  zero.  If  the  mathematical  program  cannot  find  a  feasible  solution 
to  the  problem  by  satisfying  all  constraints,  then  the  course  of  action  examined  is 
considered  insensitive.  An  insensitive  course  of  action  cannot  become  the  preferred 
alternative  regardless  of  the  weights  chosen,  e.g.  Pursue  PKK  course  of  action. 

Therefore,  it  is  an  insensitive  alternative,  and  it  is  not  possible  to  calculate  an  associated 
measure.  This  implies  the  recommendation  of  removing  the  Pursue  PKK  alternative 
because  it  will  never  become  the  top  ranked  alternative  regardless  of  the  weights  chosen 
for  the  fourteen  attributes  in  the  value  model  developed  on  2007  conditions.  Table  4.3 
shows  all  five  calculated  measures  for  the  ten  alternative  courses  of  action. 
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Table  4.3  Sensitivity  Measures 


Alternative 

Least 

Squares 

L 

norm 

oo- 

norm 

2- 

norm 

% 

Comments 

Status  Quo 

0.284 

1.181 

0.268 

0.533 

1636.80% 

Expel  AQI 

0.006 

0.150 

0.043 

0.076 

39.50% 

Train  ISI 

0.002 

0.062 

0.021 

0.039 

24.95% 

Institute  Draft 

0.447 

1.504 

0.474 

0.668 

5827.32% 

Partial  Coalition 
Withdrawal 

0.183 

0.941 

0.215 

0.428 

317.49% 

Full  Coalition 
Withdrawal 

0.120 

0.737 

0.192 

0.346 

273.08% 

Pursue  PKK 

(Insensitive  Alternative) 

Self-Sustained 

Agriculture 

0 

0 

0 

0 

0.00% 

(Ranked  #1  by  Assessed 
Weights) 

24  hr  Electricity 
Established 

0.002 

0.069 

0.025 

0.045 

48.85% 

Lower 

Unemployment 

0.003 

0.099 

0.040 

0.055 

119.55% 

Rank  ordering  these  COAs  in  ascending  order  based  on  each  measure  reveals  the 
same  ranking  as  ranking  the  COAs  in  descending  order  according  to  the  original  value, 
except  for  the  %  measure.  The  %  measure  shows  Expel  AQI  ranked  third  instead  of  fifth. 
This  ranking  irregularity  stems  from  the  fact  that  percent  change  in  weight  is  not  the 
same  as  the  absolute  change  in  weight.  Tables  4.4  -  4.8  show  the  COAs  in  rank  order 
according  to  the  measures.  There  are  comments  located  in  the  tables  output  from 
Premium  Solver  whose  explanation  follows  each  table. 
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Table  4.4  COAs  Rank  Ordered  by  the  Least  Squares  Measure 


Alternative 

Least 

Squares 

Comments 

Self-Sustained 

Agriculture 

0 

(Ranked  #1  by  Initial 
Weights) 

Train  ISI 

0.002 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

24  hr  Electricity 

Established 

0.002 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Lower  Unemployment 

0.003 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Expel  AQI 

0.006 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Full  Coalition 

Withdrawal 

0.120 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Partial  Coalition 
Withdrawal 

0.183 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Status  Quo 

0.284 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Institute  Draft 

0.447 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Pursue  PKK 

(Insensitive  Alternative) 

The  same  message  appeared  following  the  calculation  of  the  least  squares 
measure  for  each  alternative.  Using  the  Standard  GRG  Nonlinear  solver  for  these 
quadratic  programs  indicates  an  optimal  solution  that  satisfies  the  Karush-Kuhn-Tucker 
(KKT)  conditions  for  local  optimality  (Premium  Solver  V8,  2007:25 1).  Knowing  the 
problem  is  convex  reveals  that  the  local  optimal  is  a  global  optimum. 
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Table  4.5  COAs  Rank  Ordered  by  the  1-norm  Measure 


Alternative 

1-norm 

Comments 

Self-Sustained 

Agriculture 

0 

(Ranked  #1  by  Initial 
Weights) 

Train  ISI 

0.062 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

24  hr  Electricity 

Established 

0.069 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Lower  Unemployment 

0.099 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Expel  AQI 

0.150 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Full  Coalition 

Withdrawal 

0.737 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Partial  Coalition 
Withdrawal 

0.941 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Status  Quo 

1.181 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Institute  Draft 

1.504 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Pursue  PKK 

(Insensitive  Alternative) 

The  message  “Solver  found  a  solution.  All  constraints  and  optimality  conditions 
are  satisfied,”  from  the  linear  program  indicates  global  optimality  (Premium  Solver  V8, 
2007:234).  The  linear  program  cannot  find  a  combination  of  weights  that  will  lower  the 
value  of  the  measure.  A  rapid  computational  time  and  global  optimality  make  the  1-norm 
an  attractive  measure  for  future  studies. 
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Table  4.6  COAs  Rank  Ordered  by  the  oo-norm  Measure 


Alternative 

oo-norm 

Comments 

Self-Sustained 

Agriculture 

0 

(Ranked  #1  by  Initial 
Weights) 

Train  ISI 

0.021 

Solver  has  converged  to  the 
current  solution.  All 
constraints  are  satisfied. 

24  hr  Electricity 

Established 

0.025 

Solver  has  converged  to  the 
current  solution.  All 
constraints  are  satisfied. 

Lower  Unemployment 

0.040 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

Expel  AQI 

0.043 

Solver  has  converged  to  the 
current  solution.  All 
constraints  are  satisfied. 

Full  Coalition 

Withdrawal 

0.192 

Solver  has  converged  to  the 
current  solution.  All 
constraints  are  satisfied. 

Partial  Coalition 
Withdrawal 

0.215 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

Status  Quo 

0.268 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

Institute  Draft 

0.474 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

Pursue  PKK 

(Insensitive  Alternative) 

The  Standard  Evolutionary  solver  resulted  in  two  different  messages  when 
solving  for  the  infinity-norm  measure.  The  most  common  message,  “Solver  cannot 
improve  the  current  solution,”  occurs  frequently  with  this  solver  because  a  heuristic 
cannot  guarantee  optimality  (Premium  Solver  V8,  2007:236).  This  means  the  solver 
cannot  improve  the  current  solution  even  though  it  has  not  met  conditions  of  convergence 
or  optimality  (Premium  Solver  V8,  2007:235).  The  other  message,  “Solver  has 
converged  to  the  current  solution,”  indicates  that  solver  has  either  found  an  optimal 
solution  or  the  population  has  lost  diversity,  a  problem  common  in  genetic  and 
evolutionary  algorithms  (Premium  Solver  V8,  2007:257).  The  problem  prevents  the 
algorithm  from  creating  new  and  better  solutions. 
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Table  4.7  COAs  Rank  Ordered  by  the  2-norm  Measure 


Alternative 

2-norm 

Comments 

Self-Sustained 

Agriculture 

0 

(Ranked  #1  by  Initial 
Weights) 

Train  ISI 

0.039 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

24  hr  Electricity 

Established 

0.045 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Lower  Unemployment 

0.055 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Expel  AQI 

0.076 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Full  Coalition 

Withdrawal 

0.346 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Partial  Coalition 
Withdrawal 

0.428 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Status  Quo 

0.533 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Institute  Draft 

0.668 

Solver  found  a  solution.  All 
constraints  and  optimality 
conditions  are  satisfied. 

Pursue  PKK 

(Insensitive  Alternative) 

The  message  “Solver  found  a  solution.  All  constraints  and  optimality  conditions 
are  satisfied,”  for  these  quadratic  programs  indicates  an  optimal  solution  that  satisfies  the 
Karush-Kuhn-Tucker  (KKT)  conditions  for  local  optimality  (Premium  Solver  V8, 
2007:25 1).  Knowing  the  problem  is  convex  reveals  that  the  local  optimal  is  a  global 
optimum.  The  measure  values  for  the  2-norm  are  the  square  root  of  the  least  squares 
measure  values  for  every  alternative.  The  2-nonn,  however,  is  favorable  compared  to  the 
least  squares  measure  because  it  allows  for  better  visual  differentiation  between  the 
courses  of  action  when  plotted  on  a  bar  graph.  The  computational  time  between  the  two 
is  comparable. 
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Table  4.8  CO  As  Rank  Ordered  by  the  %  Measure 


Alternative 

% 

Comments 

Self-Sustained 

Agriculture 

0.00% 

(Ranked  #1  by  Initial 
Weights) 

Train  ISI 

24.95% 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

Expel  AQI 

39.50% 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

24  hr  Electricity 

Established 

48.85% 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

Lower  Unemployment 

119.55% 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

Full  Coalition 

Withdrawal 

273.08% 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

Partial  Coalition 
Withdrawal 

317.49% 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

Status  Quo 

1636.80% 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

Institute  Draft 

5827.32% 

Solver  cannot  improve  the 
current  solution.  All 
constraints  are  satisfied. 

Pursue  PKK 

(Insensitive  Alternative) 

The  Standard  Evolutionary  solver  resulted  in  the  message,  “Solver  cannot 
improve  the  current  solution,”  which  occurs  frequently  with  this  solver  because  a 
heuristic  cannot  guarantee  optimality  (Premium  Solver  V8,  2007:236).  This  means  the 
solver  cannot  improve  the  current  solution  even  though  it  has  not  met  conditions  of 
convergence  or  optimality  (Premium  Solver  V8,  2007:235). 

IV.F.  Robust  Sensitivity  Analysis 

The  calculation  of  the  measures  enables  robust  sensitivity  analysis  of  the 
alternatives.  Figure  4.3  shows  a  graphical  representation  of  the  calculated  measures.  It 
shows  a  large  relative  change  in  calculated  sensitivity  measure  between  the  top  five 
alternatives  and  the  bottom  four  from  Figure  4.2.  The  small  deltas  for  the  top  live 
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measures  indicate  sensitivity  and  that  these  alternatives  require  further  examination.  The 
rank  order  of  these  measure  values  is  the  same  for  all  measures  except  for  the  % 


Measure. 


Robust  Sensitivity  of  Alternatives 


Figure  4.3  Robust  Sensitivity  Analysis 


The  %  Measure  may  p.  The  other  measures  examine  the  absolute  distance 
between  the  assessed  weights  and  the  new  calculated  weights.  The  %  Measure  examines 
the  percent  change  for  each  weight.  According  to  this  measure,  a  change  in  weight  from 
0.01  to  0.02  and  a  change  from  0. 1  to  0.2  are  identical  with  a  100%  change.  If  a  weight 
changes  from  0.01  to  0.02  by  the  other  measures  it  is  sensitive,  while  a  change  from  0. 1 
to  0.2  does  not  indicate  sensitivity. 

Since  the  four  other  measures  indicate  the  same  rank  order,  the  recommended 
measure  is  the  1-norm.  The  1-norm  measure  has  the  shortest  computational  time  and 
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only  changes  a  few  weights.  The  short  computational  time  allows  an  analyst  to  conduct 
rapid  sensitivity  analysis  in  which  the  commander  requires  a  timely  response.  A  few 
weight  changes,  as  opposed  to  all  the  weights  changing,  could  allow  for  easier 
interpretation  of  the  weight  changes  since  it  is  sometimes  a  trade  of  weight  from  one  or 
two  attributes  to  another.  Table  4.9  shows  that  the  1-norm  measure  exchanges  weight 
between  L  H20/person/day  and  Tons  of  Supplies  Needed  in  order  to  make  Train  ISI  the 
top  alternative.  The  linear  program  transfers  the  weight  from  Tons  of  Supplies  Needed  to 
L  H20/person/day  to  give  Train  ISI  the  highest  value  of  all  alternatives. 


Table  4.9  Absolute  Change  in  Weights  by  Measures  for  Train  ISI 


Abs.  A  E 

•etween  Original  and  Calculated  Weights 

Train  ISI 

Original 

Weight 

Least 

Squares 

1-nonn 

oo-norm 

2-nonn 

% 

%  Coalition  Lost 

0.215 

0.002 

0 

0.019 

0.002 

0.053 

#  Insurgents  Crossing 

0.107 

0.001 

0 

0.002 

0.001 

0.006 

Ethno-Sectarian 
Violence  Death  Rate 

0.268 

0.002 

0 

0.006 

0.002 

0.067 

Non-Sectarian  Violent 
Death  Rate 

0.188 

0.001 

0 

0.018 

0.001 

0.025 

Avg.  hrs./day 

0.008 

0.001 

0 

0.007 

0.001 

0.002 

L  H20/person/day 

0.011 

0.017 

0.031 

0.021 

0.017 

0.003 

L  Fuel/person/day 

0.004 

0.000 

0 

0.000 

0.000 

0.001 

Tons  of  Supplies 
Needed 

0.142 

0.033 

0.031 

0.021 

0.033 

0.036 

Estimated  #  of 
Members 

0.008 

0.002 

0 

0.008 

0.002 

0.001 

Willingness 

0.016 

0.002 

0 

0.003 

0.002 

0.004 

#  of  Units  at  Level  I 

0.016 

0.004 

0 

0.020 

0.004 

0.004 

#  of  Units  at  Level  II 

0.008 

0.004 

0 

0.008 

0.004 

0.002 

#  of  Units  at  Level  III 

0.005 

0.008 

0 

0.021 

0.008 

0.001 

#  of  Units  at  Level  IV 

0.003 

0.009 

0 

0.021 

0.009 

0.001 

The  2-norm  and  Least  Squares  measures  result  in  the  calculation  of  identical  new 
weights  for  all  courses  of  action.  The  quadratic  programs  for  these  two  measures  run 
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slightly  slower  than  the  linear  programs  for  the  1-norm,  additionally,  they  cannot 
guarantee  global  optimality  and  they  change  all  weights.  Without  global  optimality 
ensured,  the  measure  could  be  greater  than  it  would  be  at  optimality  and  indicate  an 
alternative  as  insensitive  and  remove  it  from  further  consideration  when  it  was  in  fact  a 
viable  alternative  that  needs  further  examination.  When  ah  weights  change,  the 
interpretations  of  these  changes  prove  difficult  and  time  consuming.  It  is  not  as  easy  to 
see  how  the  weight  moved  from  one  attribute  to  another  as  seen  with  the  1-norm  measure 
and  time  is  often  not  a  luxury  when  trying  to  select  a  course  of  action.  The  quadratic 
programs  changing  ah  the  weights  sometime  result  in  a  weight  changing  on  a  scale  of 
0.00 1  while  another  weight  may  change  on  a  scale  of  0. 1 . 

For  example,  Table  4.10  shows  the  original  weights  and  then  the  absolute  change 
in  the  weights  required  to  raise  the  sixth  ranked  alternative,  Status  Quo,  to  make  it  the 
preferred  alternative.  When  examining  the  Least  Squares  measure,  one  can  see  that  the 
weight  for  #  of  Units  at  Level  IV  changes  by  0.003  while  the  weight  for  #  of  Units  at 
Level  II  changes  by  0.309.  This  demonstrates  two  orders  of  magnitude  difference 
between  the  changes  in  the  two  weights.  The  change  in  weight  on  a  scale  of  0. 1 
contributes  the  most  to  the  alternative’s  change  in  ranking,  while  weight  changes  on  the 
scale  of  0.001  do  not  and  could  be  artifacts  of  the  measure  optimization. 
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Table  4.10  Absolute  Change  in  Weights  by  Measures  for  Status  Quo 


Abs.  A  Between  Original  and  Calculated  Weights 

Status  Quo 

Original 

Weight 

Least 

Squares 

1-norm 

oo-norm 

2-norm 

% 

%  Coalition  Lost 

0.215 

0.176 

0.000 

0.264 

0.176 

0.361 

#  Insurgents  Crossing 

0.107 

0.107 

0.107 

0.107 

0.107 

0.107 

Ethno-Sectarian 
Violence  Death  Rate 

0.268 

0.268 

0.268 

0.268 

0.268 

0.268 

Non-Sectarian  Violent 
Death  Rate 

0.188 

0.113 

0.017 

0.186 

0.113 

0.188 

Avg.  hrs./day 

0.008 

0.008 

0.008 

0.008 

0.008 

0.008 

L  H20/person/day 

0.011 

0.011 

0.011 

0.011 

0.011 

0.011 

L  Fuel/person/day 

0.004 

0.004 

0.004 

0.004 

0.004 

0.004 

Tons  of  Supplies 
Needed 

0.142 

0.142 

0.142 

0.142 

0.142 

0.142 

Estimated  #  of 
Members 

0.008 

0.008 

0.008 

0.008 

0.008 

0.008 

Willingness 

0.016 

0.202 

0.041 

0.258 

0.202 

0.266 

#  of  Units  at  Level  I 

0.016 

0.016 

0.016 

0.015 

0.016 

0.016 

#  of  Units  at  Level  II 

0.008 

0.309 

0.550 

0.225 

0.309 

0.133 

#  of  Units  at  Level  III 

0.005 

0.005 

0.005 

0.003 

0.005 

0.005 

#  of  Units  at  Level  IV 

0.003 

0.003 

0.003 

0.001 

0.003 

0.003 

The  measures  calculated  using  non-linear  programming,  the  oo -norm  and  % 
measures,  require  more  time  than  the  quadratic  programming  measures,  and  once  again 
cannot  guarantee  optimality  and  change  all  the  weights.  The  changes  across  all  weights 
for  these  measures  show  the  same  characteristics  shown  by  the  2-norm  and  Least  Squares 
measures.  Again,  the  small  change  in  weights  could  be  artifacts  of  the  measure 
optimization. 

One-way  sensitivity  analysis  offers  a  way  to  examine  the  sensitivity  of 
alternatives  and  attribute  weights.  The  robust  sensitivity  analysis  approach  proposed  here 
examines  the  decision  sensitivity  to  the  alternatives.  Sensitive  weights  guide  a  decision 
maker  to  further  examine  the  attributes  associated  with  those  weights.  This  further 
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examination  could  include  placing  more  resources  into  soliciting  the  weight  for  the 
attribute  or  ensuring  the  evaluation  measures  are  correct  for  the  alternatives. 

A  sensitive  weight  should  have  a  common  trait  across  all  alternatives  that 
indicates  its  sensitivity.  In  one-way  sensitivity  analysis,  a  required  small  change  in 
weight  indicates  a  sensitive  alternative  and  weight  while  a  large  change  indicates  an 
insensitive  alternative  and  weight.  Does  this  same  rule  apply  to  n-way  sensitivity 
analysis  or  does  the  opposite  apply?  Does  a  required  large  change  in  weight  indicate 
sensitivity  since  small  changes  could  be  artifacts  of  the  measure  optimization?  Even  if 
the  author  knew  the  answer  to  this  question,  further  examination  could  not  find  a 
universal  trait  across  all  alternatives  for  a  weight  in  order  to  identify  a  specific  weight  or 
group  of  weights  as  sensitive.  Figure  4.4  shows  that  sensitive  alternatives  previously 
identified  shows  the  weights  for  L  H20/person/day  and  Tons  of  Supplies  Delivered 
change  the  most  for  sensitive  alternatives,  but  these  weights  change  the  least  for  non¬ 
sensitive  alternatives. 


Least  Squares  Measure 
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Figure  4.4  Absolute  Change  in  Weights  for  Least  Squares  Measure 
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Perhaps,  if  the  absolute  change  in  a  weight  ranks  in  the  top  five,  or  bottom  five, 
for  all  alternatives,  then  it  indicates  sensitivity.  Closer  examination  found  that  at  times,  a 
weight  will  remain  in  one  of  these  categories  across  all  alternatives,  but  this  is  not 
universal  across  all  measures.  Again,  even  if  it  was  universal,  is  a  ranked  weight  change 
in  the  top  or  bottom  five  classified  as  sensitive?  Additional  analysis  included  examining 
the  new  rank  order  of  weights  based  on  magnitude  compared  to  the  original  order. 
Appendix  D  shows  some  of  the  work  conducted  in  an  attempt  to  detennine  the  sensitivity 
of  the  attribute  weights  for  one  alternative  in  a  robust  sensitivity  analysis. 

IV.G.  Summary 

This  chapter  stepped  through  the  analysis  of  alternative  courses  of  action  from  a 
notional  value  model  used  to  evaluate  stability  operations  in  Iraq.  The  focus  remained  on 
the  sensitivity  analysis  of  the  model.  The  one-way  sensitivity  analysis  did  not  indicate 
sensitivity  within  the  model.  The  proposed  robust  sensitivity  analysis  approach, 
however,  indicated  that  five  of  the  ten  alternatives  are  sensitive  when  multiple  weights 
change  as  opposed  to  only  one.  Instead  of  calculating  all  five  robust  sensitivity  measures, 
the  1-norm  is  the  recommended  measure.  This  analysis  best  supports  the  initial  planning 
phases  of  military  operations  and  again  when  the  nature  of  the  problem  faced  changes. 
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V.  Discussion 


V.A.  Introduction 

Today’s  United  States  military  pursues  effects  based  operations  and  predictive 
battlespace  awareness.  If  the  military  can  predict  an  adversary’s  plans  or  can  predict  how 
an  adversary  will  react  to  actions  by  the  United  States,  then  the  United  States  military  can 
increase  its  effectiveness.  This  is  not  a  new  concept.  Military  commanders  have 
recognized  the  need  for  this  capability  at  least  as  early  as  Sun  Tzu  in  the  sixth  century 
B.C.  In  the  face  of  this  capability  gap,  commanders  desire  implementable  courses  of 
action  that  are  robust  to  uncertainty. 

One  approach  to  assessing  courses  of  action  used  in  today’s  military  is  additive 
value  models.  The  problem  lies  within  the  solicitation  of  weights.  The  weights  are 
subjective  assessments  of  importance  between  the  attributes  in  the  model.  When 
evaluating  courses  of  action,  an  ideal  course  of  action  dominates  all  others.  A  dominant 
alternative  scores  better  in  every  evaluation  measure  compared  to  other  alternatives.  If  a 
course  of  action  is  dominant,  then  the  value  model  is  insensitive.  There  is  rarely  a 
dominant  course  of  action  to  select. 

If  there  is  not  a  dominant  solution,  then  the  commander  desires  a  robust 
alternative  that  is  relatively  insensitive  and  still  meets  his/her  objectives.  Insensitive 
alternatives  indicate  a  required  large  change  in  the  weights  in  order  for  those  alternatives 
to  become  the  preferred  alternative.  If  a  large  change  in  weights  is  required  to  cause  a 
change,  then  the  assessed  weights  can  be  slightly  incorrect  and  the  recommended 
decision  will  not  change. 
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V.B.  Research  Contributions 


This  research  contributes  to  the  field  of  sensitivity  analysis  for  additive  value 
models.  It  not  only  compares  the  results  from  three  previously  published  sensitivity 
measures,  but  it  also  proposes  a  new  sensitivity  measure  that  is  not  recommended.  These 
sensitivity  measures  enable  a  more  robust  approach  to  sensitivity  than  the  standard  one¬ 
way  sensitivity  analysis.  Instead  of  varying  one  weight  at  time,  the  robust  approach  using 
sensitivity  measures  allows  for  the  interaction  of  weight  changes.  This  approach  can 
indicate  sensitive  alternatives  requiring  further  examination. 

V.C.  Recommendations  for  Further  Research 

After  discovering  sensitive  alternatives,  closer  examination  should  occur  on 
sensitive  attributes.  One-way  sensitivity  analysis  indicates  sensitive  attributes  and  then 
indicates  the  alternatives  sensitive  to  that  attribute.  This  approach  to  robust  sensitivity 
analysis  works  in  the  opposite  order.  First,  it  identifies  sensitive  alternatives.  Then,  it 
would  identify  the  sensitive  attributes.  Detennining  these  sensitive  attributes  is  an 
avenue  for  future  research.  Traditional  sensitivity  analysis  and  the  robust  sensitivity 
analysis  for  alternatives  indicate  sensitivity  based  on  small  changes  in  weights.  When 
trying  to  detennine  the  sensitive  attributes  using  the  robust  sensitivity  approach,  does  a 
required  small  or  large  change  in  an  attribute  weight  indicate  sensitivity?  Another 
avenue  includes  testing  these  measures  on  several  additive  value  models  to  find 
consistency.  The  notional  test  case  in  this  research  was  intentionally  insensitive  on  all 
attributes  according  to  one-way  sensitivity  analysis.  How  does  the  outcome  of  the 
sensitivity  measures  change  when  there  is  an  indication  of  sensitivity  in  the  model  based 
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on  one  attribute?  An  additional  avenue  for  further  research  includes  examining  the 
sensitivity  of  a  decision  based  on  the  SDVFs  and  evaluation  measures. 

V.D.  Conclusions 

Robust  sensitivity  analysis  using  sensitivity  measures  provides  a  more  realistic 
approach  to  sensitivity  analysis  for  additive  value  models  in  the  planning  stage.  The 
world  constantly  changes,  and  generally  does  not  change  only  one  attribute  at  a  time. 
Instead,  multiple  attributes  change  simultaneously.  The  robust  sensitivity  measures 
attempt  to  address  this  situation. 
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Appendix  A.  Evaluation  Measure  Definitions 


Table  A.l  Evaluation  Measure  Summary 


Level  I:  Fully  Independent  (Petraeus,  2007: 12) 

Level  II:  Iraqi  Lead  with  Coalition  Support  (Petraeus,  2007:12) 
Level  III:  Fighting  Side  by  Side  (Petraeus,  2007:12) 

Level  IV:  Unit  Forming  (Petraeus,  2007:12) 
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Appendix  B.  Single  Dimensional  Value  Functions 


Value 


Figure  B.l  Coalition  Forces’  Safety  SDVF 


Value 


Figure  B.2  Estimated  Insurgent  Border  Crossings  SDVF 
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Value 


Ethno-Sectarian  Violence  Death  Rate  (Yearly  deaths/ 100,000) 


Figure  B.3  Ethno-Sectarian  Violence  SDVF 


Value 


Non-Sectarian  Violent  Death  Rate  (Yearly  deaths/ 100,000) 


Figure  B.4  Non-Sectarian  Violence  SDVF 
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Value 


0. 


Avg.  hrs./day  (Hours/day) 


24. 


Figure  B.5  Electricity  SDVF 


Value 


Figure  B.6  Fuel  (Heating/Cooking)  SDVF 
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1 


Value 


Figure  B.7  Potable  Water  SDVF 


Value 


Figure  B.8  Flumanitarian  Relief  SDVF 


76 


Label 

High 

Medium 

Low 


Value 


Figure  B.9  Additional  People  Needed  for  ISI  SDVF 


Value 

1.00 

0.50 

0.00 


Figure  B.10  Locals’  Willingness  to  Serve  SDVF 
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Value 


Figure  B.  1 1  Capability  Level  I  SDVF 


Value 


Figure  B.12  Capability  Level  II  SDVF 
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Value 


Figure  B.13  Capability  Level  III  SDVF 


Value 


Figure  B.14  Capability  Level  IV  SDVF 
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Appendix  C.  One-Way  Sensitivity  Analysis 
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Figure  C.l  Coalition  Forces’  Safety  One-Way  Sensitivity  Graph 
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Figure  C.2  Estimated  Insurgent  Border  Crossings  One-Way  Sensitivity  Graph 


80 


Self-Sustained 

Train 

24  hr  Electricity 
Lower 
Expel 
Status 

Partial  Coalition 
Institute 
Full  Coalition 
Pursue 


Percent  of  Weight  on  Ethno-Sectarian  Violence  Death  Rate  Measure 

Figure  C.3  Ethno-Sectarian  Violence  One-Way  Sensitivity  Graph 
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Figure  C.4  Non-Sectarian  Violence  One-Way  Sensitivity  Graph 
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Figure  C.5  Electricity  One-Way  Sensitivity  Graph 
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Figure  C.6  Fuel  (Fleating/Cooking)  One-Way  Sensitivity  Graph 
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Figure  C.7  Potable  Water  One-Way  Sensitivity  Graph 
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Figure  C.8  Humanitarian  Relief  One-Way  Sensitivity  Graph 
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Figure  C.9  Additional  People  Needed  for  ISI  One-Way  Sensitivity  Graph 
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Figure  C.10  Locals’  Willingness  to  Serve  One-Way  Sensitivity  Graph 
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Figure  C.l  1  Capability  Level  I  One-Way  Sensitivity  Graph 
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Figure  C.12  Capability  Level  II  One-Way  Sensitivity  Graph 
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Figure  C.13  Capability  Level  III  One-Way  Sensitivity  Graph 
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Figure  C.14  Capability  Level  IV  One-Way  Sensitivity  Graph 
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Appendix  D.  Robust  Sensitivity  Analysis  of  Attributes 

A  small  change  in  a  single  weight  using  one-way  sensitivity  analysis  indicates 
sensitive  alternatives  and  the  weight  they  are  sensitive  to.  Looking  at  the  robust 
sensitivity  measures  described  in  this  research  presents  a  new  problem.  Is  it  a  large  or 
small  change  in  a  weight  needed  for  an  alternative  to  become  preferred  that  indicates 
sensitivity?  The  tables  and  diagrams  below  show  a  few  approaches  used  to  attempt  to 
answer  this  question.  Table  D.l  shows  the  rank  order  of  the  absolute  change  in  weight 
across  each  measure.  Figure  D.l  shows  the  information  from  Table  D.  1  as  a  contour  plot 
breaking  the  ranks  into  the  top,  middle,  and  bottom  five  weights.  Table  D.2  looks  at  the 
rank  order  of  the  calculated  weights  for  each  measure  compared  to  the  rank  order  of  the 
initial  weights  from  the  value  model. 


Table  D.  1  Rank  Order  of  the  Change  in  Weights  for  Train  ISI 


Train  ISI 

Least  Squares 

1-nonn 

co-norm 

2 -norm 

% 

Tons  of  Supplies  Needed 

1 

2 

4 

1 

3 

L  H20/person/day 

2 

1 

3 

2 

8 

#  of  Units  at  Level  IV 

3 

3 

1 

3 

13 

#  of  Units  at  Level  III 

4 

3 

2 

4 

11 

#  of  Units  at  Level  II 

5 

3 

8 

5 

9 

#  of  Units  at  Level  I 

6 

3 

5 

6 

7 

Willingness 

7 

3 

12 

7 

6 

Ethno-Sectarian  Violence 
Death  Rate 

8 

3 

11 

8 

1 

%  Coalition  Lost 

9 

3 

6 

9 

2 

Estimated  #  of  Members 

10 

3 

9 

10 

12 

#  Insurgents  Crossing 

11 

3 

13 

11 

5 

Non-Sectarian  Violent  Death 
Rate 

12 

3 

7 

12 

4 

Avg.  hrs./day 

13 

3 

10 

13 

10 

L  Fuel/person/day 

14 

3 

14 

14 

14 

87 


Train  ISI 


Figure  D.  1  3D  Bar  Graph  of  Data  from  Table  D.  1 


Table  D.2  Descending  Rank  Order  of  Weights  for  Train  ISI 


Train  ISI 

Swing 

Weighting 

Least 

Squares 

1-Norm 

oo-Norm 

2 -Norm 

% 

Ethno-Sectarian 
Violence  Death 
Rate 

0.268 

0.271 

0.268 

0.285 

0.271 

0.335 

%  Coalition 
Lost 

0.215 

0.212 

0.215 

0.193 

0.212 

0.161 

Non-Sectarian 
Violent  Death 
Rate 

0.188 

0.189 

0.188 

0.167 

0.189 

0.234 

Tons  of 
Supplies 
Needed 

0.142 

0.110 

0.111 

0.121 

0.110 

0.107 

#  Insurgents 
Crossing 

0.107 

0.108 

0.107 

0.093 

0.108 

0.082 

Willingness 

0.016 

0.014 

0.016 

0.021 

0.014 

0.012 

#  of  Units  at 
Level  I 

0.016 

0.020 

0.016 

0.034 

0.020 

0.020 

L 

H20/person/day 

0.011 

0.029 

0.042 

0.033 

0.029 

0.014 

Estimated  #  of 
Members 

0.008 

0.006 

0.008 

0.000 

0.006 

0.007 

#  of  Units  at 
Level  II 

0.008 

0.005 

0.008 

0.000 

0.005 

0.006 

Avg.  hrs./day 

0.008 

0.008 

0.008 

0.002 

0.008 

0.009 

#  of  Units  at 
Level  III 

0.005 

0.014 

0.005 

0.027 

0.014 

0.007 

L 

Fuel/person/day 

0.004 

0.004 

0.004 

0.001 

0.004 

0.004 

#  of  Units  at 
Level  IV 

0.003 

0.011 

0.003 

0.024 

0.011 

0.003 

89 


Bibliography 


AFC2ISRC/A3.  (2006).  Capability  development  document  for  air  and  space  operations 
center  (AOC)  weapon  system  modernization 

Alloway,  J.  A., Jr.  (1997).  Be  prepared  with  an  affinity  diagram.  Quality  Progress,  30(7), 
75-77. 


Barron,  H.,  &  Schmidt,  C.  P.  (1988).  Sensitivity  analysis  of  additive  multiattribute  value 
model.  Operations  Research,  36(1),  122. 

Bassham,  B.,  Bauer,  K.  W.,  &  Miller,  J.  O.  (2006).  Automatic  target  recognition  system 
evaluation  using  decision  analysis  techniques.  Military  Operations  Research,  11(1), 
49-66. 

Bauer,  Kenneth.  Professor,  Air  Force  Institute  of  Technology,  Dayton  OH.  Personal 
Correspondence.  January  2008. 

Bauer,  K.  W.,Jr.,  Parnell,  G.  S.,  &  Meyers,  D.  A.  (1999).  Response  surface  methodology 
as  a  sensitivity  analysis  tool  in  decision  analysis.  Journal  of  Multicriteria  Decision 
Analysis,  5(3),  162. 

Bednarski,  M.,  Cholewa,  W.,  &  Frid,  W.  (2003).  The  sensitivity  analysis  of  bayesian 
networks.  Al-METH  2003,  Symposium  on  Methods  of  Artificial  Intelligence,  ,  29-32. 

Brassard,  M.,  Ritter,  D.,  &  Goal/qpc.  (1994).  The  memory  jogger  II :  A  pocket  guide  of 
tools  for  continuous  improvement  &  effective  planning  (1st  ed.).  Methuen,  Mass.: 
Goal/qpc. 

Chambal,  S.,  Shoviak,  M.,  &  Thai,  A.  E.,Jr.  (2003).  Decision  analysis  methodology  to 
evaluate  integrated  solid  waste  management  alternatives.  Environmental  Modeling  & 
Assessment,  5(1),  25. 

Clemen,  R.  T.,  Reilly,  T.,  &  Clemen,  R.  T.  (2001).  Making  hard  decisions  with 
DecisionTools  (2nd  rev.  ed.).  Pacific  Grove,  CA:  Duxbury/Thomson  Learning. 

Coram,  R.  (2002).  Boyd  :  The  fighter  pilot  who  changed  the  art  of  war  (1st  ed.).  Boston: 
Little,  Brown. 

Doyle,  M.  P.,  Deckro,  R.  F.,  Kloeber,  J.  M.,  &  Jackson,  J.  A.  (2000).  Measures  of  merit 
for  offensive  infonnation  operations  courses  of  action.  Military  Operations  Research, 
5(2),  5. 


90 


Ewing,  P.  L.,Jr.,  Tarantino,  W.,  &  Parnell,  G.  S.  (2006).  Use  of  decision  analysis  in  the 
army  base  realignment  and  closure  ( BRAG )  2005  military  value  analysis 

Factsheets  :  Air  force  research  laboratory  :  Air  force  research  laboratory.  Retrieved 
10/30/2007,  2007,  from  http://www.ar.mil/factshccts/ractshcct.asp7icR148 

Fensterer,  G.  D.  Planning  and  assessing  stability  operations:  A  proposed  value  focus 
thinking  approach.  MS  Thesis,  AFIT/GOR/ENS/07-06,  Air  Force  Institute  of 
Technology  (AU),  Wright-Patterson  AFB  OH,  March  2007  (ADA466687)). 

Frontline  Systems  (2007).  Premium  Solver  Platform  Version  8.0  User  Guide.  Incline 
Village,  NV:  Frontline  Systems. 

Helton,  J.  C.  (1993).  Uncertainty  and  sensitivity  analysis  techniques  for  use  in 

performance  assessment  for  radioactive  waste  disposal.  Reliability  Engineering  & 
System  Safety,  42(2-3),  327-367. 

Hughes,  C.,  &  Hughes,  K.  K.  (1990).  N-way  sensitivity  analysis  -  a  verification  and 
stability-assessment  technique  for  completely  subjective  decision- analysis  trees. 
Medical  Decision  Making,  10(1),  68-74. 

Kahraman,  Y.  R.  Robust  sensitivity  analysis  for  multi-attribute  deterministic  hierarchical 
value  models.  MS  Thesis,  AFIT/GOR/ENS/02-10,  Air  Force  Institute  of  Technology 
(AU),  Wright-Patterson  AFB  OH,  March  2002  (ADA400032). 

Keeney,  R.  F.  (1992).  Value-focused  thinking  :  A  path  to  creative  decisionmaking. 
Cambridge,  Mass.:  Harvard  University  Press. 

Kerchner,  P.  M.,  Deckro,  R.  F.,  &  Kloeber,  J.  M.  (2001).  Valuing  psychological 
operations.  Military  Operations  Research,  6(2),  45. 

Kirkwood,  C.  W.  (1997).  Strategic  decision  making  :  Multiobjective  decision  analysis 
with  spreadsheets.  Belmont:  Duxbury  Press. 

Parnell,  G.  S.  (2007).  Value-focused  thinking  using  multiple  objective  decision  analysis. 
In  A.  G.  Foerch,  &  F.  B.  Rainey  (Eds.),  Methods  for  conducting  military  operational 
analysis:  Best  practices  in  use  throughout  the  department  of  defense  (1st  ed.,  ). 
Alexandria,  VA:  Military  Operations  Research  Society. 

Parnell,  G.  S.,  Conley,  H.  W.,  Jackson,  J.  A.,  Fehmkuhl,  F.  J.,  &  Andrew,  J.  M.  (1998). 
Foundations  2025:  A  value  model  for  evaluating  future  air  and  space  forces. 
Management  Science,  44(10),  1336-1350. 

Phipps,  K.  K.  Value-focused  thinking  and  course  of  action  selection.  MS  Thesis.  The 
Graduate  School  at  the  University  of  Nebraska,  Fincoln  NE,  2006. 


91 


Pruitt,  A.  K.  Modeling  homeland  security:  A  value  focused  thinking  approach.  MS 
Thesis,  AFIT/GOR/ENS/03-19,  Air  Force  Institute  of  Technology  (AU),  Wright- 
Patterson  AFB  OH,  March  2003  (ADA413873). 

Petraeus,  D.  H.,  Commander,  Multi-National  Force-Iraq.  Report  to  Congress  on  the 
Situation  in  Iraq.  10-11  September  2007. 

Ringuest,  J.  L.  (1997).  L-p-metric  sensitivity  analysis  for  single  and  multi-attribute 
decision  analysis.  European  Journal  of  Operational  Research,  95(3),  563-570. 

Rios  Insua,  D.,  &  French,  S.  (1991).  Framework  for  sensitivity  analysis  in  discrete  multi¬ 
objective  decision-making.  European  Journal  of  Operational  Research,  54(2),  176- 
190. 

Robbins,  M.  J.  Investigating  the  complexities  of  nationbuilding:  A  sub-national  regional 
perspective.  MS  Thesis,  AFIT/GOR/ENS/05-16,  Air  Force  Institute  of  Technology 
(AU),  Wright-Patterson  AFB  OH,  March  2005  (ADA435214). 

SCIENTIFIC  ADVISORY  BOARD  (AIR  FORCE)  WASHINGTON  DC,  Harrison,  G. 

B.,  Johnson,  R.  O.,  O’Neill,  M.  R.,  Ganz,  M.  W.,  &  Saunders,  T.  F.  (2002).  (U) 
predictive  battlespace  awareness  to  improve  military  effectiveness,  volume  L 
summary 

Scupin,  R.  (1997).  The  KJ  method:  A  technique  for  analyzing  data  derived  from  japanese 
ethnology.  Human  Organization,  56(2),  233. 

Shoviak,  M.  J.  (2001).  Decision  analysis  methodology  to  evaluate  integrated  solid  waste 
management  alternatives  for  a  remote  alaskan  air  station.  MS  Thesis, 
AFIT/GEE/ENV/01M-20,  Air  Force  Institute  of  Technology  (AU),  Wright-Patterson 
AFB  OH,  March  2001  (AD A390853). 

Stewart,  T.  J.  (1996).  Robustness  of  additive  value  function  methods  in  MCDM.  Journal 
of  Multi-Criteria  Decision  Analysis,  5(4),  301-309. 

Sunzi,  &  Griffith,  S.  B.  (2005).  The  illustrated  art  of  war  [Sunzi  bing  fa.]  .  Oxford  ;  New 
York:  Oxford  University  Press. 

US  Department  of  Defense.  (2006).  Joint  operation  planning  No.  JP5-0) 

US  Department  of  Defense.  (2006).  Joint  operations  No.  JP3-0) 

US  Department  of  Defense.  (2007).  Doctrine  for  the  armed  forces  of  the  united  states  No. 
JP1) 


92 


Von  Winterfeldt,  D.,  &  Edwards,  W.  (1986).  Decision  analysis  and  behavioral  research. 
Cambridge  Cambridgeshire  ;  New  York:  Cambridge  University  Press. 

Wolters,  W.  T.  M.,  &  Mareschal,  B.  (1995).  Novel  types  of  sensitivity  analysis  for 
additive  MCDM  methods.  European  Journal  of  Operational  Research,  81(2),  281- 
290. 

Wu,  W.,  Chou,  C.  H.,  &  Wu,  Y.  (2004).  A  study  of  strategy  implementation  as  expressed 
through  sun  tzu's  principles  of  war.  Industrial  Management  and  Data  Systems, 
104(5),  396-408.  http://dx.doi.org/10.1108/0263557041053748Q: 
http://dx.doi.org/10.1108/02635570410537480 


93 


Vita 


Hunter  A.  Marks  graduated  from  St.  Amant  High  School  in  St.  Amant,  Louisiana. 
He  entered  undergraduate  studies  at  the  United  States  Air  Force  Academy  in  Colorado 
Springs,  Colorado  where  he  completed  Basic  Cadet  Training  and  one  semester  before 
being  medically  dis-enrolled  in  December  2001.  He  resumed  his  undergraduate  studies 
at  Louisiana  State  University  in  Baton  Rouge,  Louisiana  where  he  graduated  with  a 
Bachelor  of  Science  degree  in  Industrial  Engineering  in  May  2005.  He  entered  civil 
service  through  the  Air  Force  PALACE  Acquire  Internship  Program. 

His  first  assignment  was  at  Offutt  Air  Force  Base,  Nebraska  as  an  Operations 
Research  Analyst  for  the  United  States  Strategic  Command  in  August  2005.  In  August 
2006,  he  entered  the  Graduate  School  of  Engineering  and  Management,  Air  Force 
Institute  of  Technology.  Upon  graduation,  he  will  return  to  the  United  States  Strategic 
Command. 


94 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  074-0188 

The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  the  collection  of  information,  including 
suggestions  for  reducing  this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway, 

Suite  1204,  Arlington,  VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  an  penalty  for  failing  to  comply  with  a  collection  of 
information  if  it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 

1 .  REPORT  DATE  (DD-MM-YYYY)  2.  REPORT  TYPE 

03-01-2008  Master’s  Thesis 

3.  DATES  COVERED  (From-  To) 

May  2007  -  Mar  2008 

4.  TITLE  AND  SUBTITLE 

ROBUST  SENSITIVITY  ANALYSIS  OF  COURSES  OF  ACTION 
USING  AN  ADDITIVE  VALUE  MODEL 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

Marks,  Flunter,  A.,  Civilian,  USAF 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAMES(S)  AND  ADDRESS(S) 

Air  Force  Institute  of  Technology 

Graduate  School  of  Engineering  and  Management  (AFIT/EN) 

2950  Hobson  Street,  Building  642 

WPAFB  OH  45433-7765 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

AFIT  /  GOR/EN  S/08-14 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Research  Laboratories/Information  Directorate  (AFRL/RISE) 
Attn:  Mr.  Jerry  L.  Dussault 

525  Brooks  Rd.  DSN:  587-2067 

Rome,  NY  13441-4505  e-mail:  Jerry.Dussault@rl.af.mil 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

The  Department  of  Defense  (DoD)  requires  the  ability  to  quantifiably  measure  progress  in  arenas  that  are  complex  and  difficult  to  measure,  such  as  the  stability  of  a 

region.  Therefore,  the  DoD  works  diligently  to  predict  the  effect  of  operations  and  sponsors  research  to  improve  prediction  and  analysis.  They  desire  a  repeatable, 
systematic  methodology  to  aid  in  the  selection  of  courses  of  action  (CO A)  that  efficiently  meet  stated  objectives  and  quantitatively  measure  the  degree  of 
accomplishment  of  these  objectives.  The  author  proposes  a  value-focused  thinking  (VFT)  decision  analysis  (DA)  approach  to  this  problem.  This  methodology  not 
only  aids  in  selection  of  possible  COAs,  but  provides  a  framework  to  compare  the  effectiveness  of  implemented  actions  via  key  indicators.  Due  to  the  complex  nature 
of  COA  selection  and  assessment,  weights  within  the  DA  model  are  often  fluid.  Sensitivity  analysis  provides  the  justification  of  COA  selection  in  such  an 
environment.  This  thesis  focuses  on  conducting  further  analysis  of  the  ranked  alternatives  through  a  robust  sensitivity  analysis  technique. 

Sensitivity  analysis  begins  with  the  examination  of  the  top  ranked  alternative  by  varying  one  weight  at  a  time,  one-way  sensitivity.  The  author  then  proposes  a  more 
robust  examination  of  multiple  weight  sensitivity  using  five  unique  measures  and  optimization  via  linear  and  non-linear  programming.  The  measures  reveal  the 
alternatives  sensitive  to  small  simultaneous  variations  of  multiple  weights  within  the  model,  n-way  sensitivity.  Small  measure  values  indicate  sensitive  alternatives, 
and  indicate  to  a  field  commander  where  to  more  closely  examine  the  consequences  of  a  selected  COA. 

15.  SUBJECT  TERMS 

Robust  Sensitivity  Analysis,  Course  of  Action  selection,  n-Way  Sensitivity  Analysis,  Affinity  Diagramming, 
Additive  Value  Model,  Stability  Operations 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

18.  NUMBER 

19a.  NAME  OF  RESPONSIBLE  PERSON 

ABSTRACT 

OF 

Shane  A.  Knighton,  Maj,  USAF  (ENS) 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

PAGES 

19b.  TELEPHONE  NUMBER  (Include  area  code) 

u 

u 

u 

UU 

108 

(937)  255-3636,  ext  4575;  e-mail:  Shane. KnishtonOufit.edu 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  Z39-18 


